2023-10-13T23:52:17.765 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-13T23:52:17.766 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_dis@teuthology --name dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi --archive /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279 --description rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --verbose -- /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279/orig.config.yaml 2023-10-13T23:52:17.800 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-13T23:52:17.891 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279 branch: wip-dis-testing description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: idryomov@gmail.com first_in_suite: false job_id: '7427279' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-qa sha1: 735f68d7bf91d643813a8420fc2b11441f4bd8c4 owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2449 sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f sleep_before_teardown: 0 subset: 49/99 suite: rbd suite_branch: wip-dis-testing-qa suite_path: /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: 735f68d7bf91d643813a8420fc2b11441f4bd8c4 targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGpmrSRWtaEDZQ9e8lI3BFxZn+4PtOyoHSrzMszzz2knv/8eHKNBninwN6j44sm9RCFUamsrbYa94st1WdqRN4= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-13_23:35:38 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-13T23:52:17.891 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa; will attempt to use it 2023-10-13T23:52:17.892 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks 2023-10-13T23:52:17.893 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-13T23:52:17.894 INFO:teuthology.task.internal:Checking packages... 2023-10-13T23:52:17.918 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '01fff6a72a328459c1d153e5dc1de6a34e48a82f' 2023-10-13T23:52:17.919 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-13T23:52:17.919 INFO:teuthology.packaging:ref: None 2023-10-13T23:52:17.919 INFO:teuthology.packaging:tag: None 2023-10-13T23:52:17.920 INFO:teuthology.packaging:branch: wip-dis-testing 2023-10-13T23:52:17.920 INFO:teuthology.packaging:sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:52:17.920 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2023-10-13T23:52:18.070 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:52:18.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-13T23:52:18.077 INFO:teuthology.task.internal:no buildpackages task found 2023-10-13T23:52:18.077 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-13T23:52:18.089 INFO:teuthology.task.internal:Saving configuration 2023-10-13T23:52:18.101 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-13T23:52:18.108 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-13T23:52:18.134 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-13 23:46:43.236397', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGs3xr7gzKJ0O1xpHNKtINBRGeRsK7/PrjsX709N3QWk'} 2023-10-13T23:52:18.135 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-13T23:52:18.139 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-13T23:52:18.140 INFO:teuthology.run_tasks:Running task console_log... 2023-10-13T23:52:18.184 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f44643a8940>, signals=[15]) 2023-10-13T23:52:18.185 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-13T23:52:18.191 INFO:teuthology.task.internal:Opening connections... 2023-10-13T23:52:18.191 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2023-10-13T23:52:18.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:52:18.271 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-13T23:52:18.277 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2023-10-13T23:52:18.374 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2023-10-13T23:52:18.375 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2023-10-13T23:52:18.421 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-13T23:52:18.421 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Ubuntu" 2023-10-13T23:52:18.421 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="22.04" 2023-10-13T23:52:18.421 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-13T23:52:18.422 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_CODENAME=jammy 2023-10-13T23:52:18.422 INFO:teuthology.orchestra.run.smithi081.stdout:ID=ubuntu 2023-10-13T23:52:18.422 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE=debian 2023-10-13T23:52:18.422 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-13T23:52:18.422 INFO:teuthology.orchestra.run.smithi081.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-13T23:52:18.423 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-13T23:52:18.423 INFO:teuthology.orchestra.run.smithi081.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-13T23:52:18.423 INFO:teuthology.orchestra.run.smithi081.stdout:UBUNTU_CODENAME=jammy 2023-10-13T23:52:18.424 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2023-10-13T23:52:18.445 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-13T23:52:18.453 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-13T23:52:18.458 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-13T23:52:18.459 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2023-10-13T23:52:18.464 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-13T23:52:18.470 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-13T23:52:18.470 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2023-10-13T23:52:18.511 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-13T23:52:18.543 INFO:teuthology.run_tasks:Running task kernel... 2023-10-13T23:52:18.558 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-13T23:52:18.559 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-13T23:52:18.559 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-13T23:52:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:52:18.564 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2023-10-13T23:52:18.610 INFO:teuthology.orchestra.run.smithi081.stdout:5.15.0-56-generic 2023-10-13T23:52:18.610 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.15.0-56-generic 2023-10-13T23:52:18.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get clean 2023-10-13T23:52:18.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2023-10-13T23:52:18.876 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:52:18.898 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-13T23:52:18.961 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-13T23:52:18.997 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-13T23:52:19.105 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,062 kB] 2023-10-13T23:52:19.185 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [506 kB] 2023-10-13T23:52:19.190 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [234 kB] 2023-10-13T23:52:19.192 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-13T23:52:19.192 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [974 kB] 2023-10-13T23:52:19.207 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-13T23:52:19.207 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [157 kB] 2023-10-13T23:52:19.208 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-13T23:52:19.209 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [991 kB] 2023-10-13T23:52:19.217 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [658 kB] 2023-10-13T23:52:19.241 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [217 kB] 2023-10-13T23:52:19.243 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-13T23:52:19.243 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-13T23:52:19.243 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-13T23:52:19.244 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-13T23:52:19.244 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-13T23:52:19.245 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-13T23:52:19.245 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-13T23:52:19.245 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-13T23:52:19.259 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-13T23:52:19.276 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-13T23:52:19.276 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-13T23:52:19.277 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-13T23:52:19.277 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-13T23:52:19.422 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [346 kB] 2023-10-13T23:52:19.682 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [856 kB] 2023-10-13T23:52:19.826 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [175 kB] 2023-10-13T23:52:19.846 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-13T23:52:19.847 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-13T23:52:19.849 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [953 kB] 2023-10-13T23:52:19.912 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [154 kB] 2023-10-13T23:52:19.922 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-13T23:52:19.922 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [560 kB] 2023-10-13T23:52:19.948 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [789 kB] 2023-10-13T23:52:20.138 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-13T23:52:20.522 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-10-13T23:52:20.529 INFO:teuthology.orchestra.run.smithi081.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-13T23:52:20.530 INFO:teuthology.orchestra.run.smithi081.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-13T23:52:20.544 INFO:teuthology.orchestra.run.smithi081.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-13T23:52:20.548 INFO:teuthology.orchestra.run.smithi081.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-13T23:52:25.046 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 9,489 kB in 2s (5,482 kB/s) 2023-10-13T23:52:26.300 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-13T23:52:26.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-13T23:52:26.368 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-13T23:52:26.531 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2023-10-13T23:52:26.531 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2023-10-13T23:52:26.670 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:52:26.670 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:52:26.670 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:52:26.671 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:52:26.671 INFO:teuthology.orchestra.run.smithi081.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-13T23:52:26.671 INFO:teuthology.orchestra.run.smithi081.stdout: python2.7-minimal 2023-10-13T23:52:26.672 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:52:26.672 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2023-10-13T23:52:26.672 INFO:teuthology.orchestra.run.smithi081.stdout: linux-generic linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-13T23:52:26.673 INFO:teuthology.orchestra.run.smithi081.stdout: linux-headers-generic linux-image-5.15.0-86-generic 2023-10-13T23:52:26.673 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-5.15.0-86-generic linux-modules-extra-5.15.0-86-generic 2023-10-13T23:52:26.673 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2023-10-13T23:52:26.673 INFO:teuthology.orchestra.run.smithi081.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-13T23:52:26.674 INFO:teuthology.orchestra.run.smithi081.stdout:Recommended packages: 2023-10-13T23:52:26.674 INFO:teuthology.orchestra.run.smithi081.stdout: thermald 2023-10-13T23:52:26.718 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2023-10-13T23:52:26.718 INFO:teuthology.orchestra.run.smithi081.stdout: linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-13T23:52:26.718 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.15.0-86-generic linux-modules-5.15.0-86-generic 2023-10-13T23:52:26.719 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-extra-5.15.0-86-generic 2023-10-13T23:52:26.719 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages will be upgraded: 2023-10-13T23:52:26.720 INFO:teuthology.orchestra.run.smithi081.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-13T23:52:26.768 INFO:teuthology.orchestra.run.smithi081.stdout:3 upgraded, 5 newly installed, 0 to remove and 268 not upgraded. 2023-10-13T23:52:26.768 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 113 MB of archives. 2023-10-13T23:52:26.768 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-13T23:52:26.769 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-86-generic amd64 5.15.0-86.96 [22.4 MB] 2023-10-13T23:52:27.176 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-86-generic amd64 5.15.0-86.96 [11.5 MB] 2023-10-13T23:52:27.265 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-86-generic amd64 5.15.0-86.96 [63.9 MB] 2023-10-13T23:52:28.078 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.86.83 [1,702 B] 2023-10-13T23:52:28.111 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.86.83 [2,472 B] 2023-10-13T23:52:28.116 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86 all 5.15.0-86.96 [12.3 MB] 2023-10-13T23:52:28.265 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86-generic amd64 5.15.0-86.96 [2,882 kB] 2023-10-13T23:52:28.379 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.86.83 [2,328 B] 2023-10-13T23:52:28.752 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 113 MB in 2s (68.4 MB/s) 2023-10-13T23:52:28.952 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-modules-5.15.0-86-generic. 2023-10-13T23:52:31.003 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-13T23:52:31.008 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../0-linux-modules-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:31.091 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:33.476 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-image-5.15.0-86-generic. 2023-10-13T23:52:33.498 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../1-linux-image-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:33.588 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:34.421 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-86-generic. 2023-10-13T23:52:34.442 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:34.479 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:40.940 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../3-linux-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:52:41.083 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:52:41.366 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:52:41.462 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:52:41.773 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-headers-5.15.0-86. 2023-10-13T23:52:41.795 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../5-linux-headers-5.15.0-86_5.15.0-86.96_all.deb ... 2023-10-13T23:52:41.840 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-13T23:52:47.356 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-headers-5.15.0-86-generic. 2023-10-13T23:52:47.393 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../6-linux-headers-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:47.439 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:50.751 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:52:50.877 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-headers-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:52:51.211 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-13T23:52:51.332 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:51.477 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-headers-generic (5.15.0.86.83) ... 2023-10-13T23:52:51.611 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:52.952 INFO:teuthology.orchestra.run.smithi081.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:52:52.952 INFO:teuthology.orchestra.run.smithi081.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-13T23:52:52.952 INFO:teuthology.orchestra.run.smithi081.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:52:52.952 INFO:teuthology.orchestra.run.smithi081.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-86-generic 2023-10-13T23:52:53.115 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:54.907 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:56.325 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-generic (5.15.0.86.83) ... 2023-10-13T23:52:56.443 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-generic (5.15.0.86.83) ... 2023-10-13T23:52:56.561 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:56.624 INFO:teuthology.orchestra.run.smithi081.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-13T23:52:56.624 INFO:teuthology.orchestra.run.smithi081.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:53:11.162 INFO:teuthology.orchestra.run.smithi081.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-13T23:53:11.162 INFO:teuthology.orchestra.run.smithi081.stdout:Sourcing file `/etc/default/grub' 2023-10-13T23:53:11.181 INFO:teuthology.orchestra.run.smithi081.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-13T23:53:11.198 INFO:teuthology.orchestra.run.smithi081.stdout:Generating grub configuration file ... 2023-10-13T23:53:11.548 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:53:11.571 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:53:11.899 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:53:11.904 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-13T23:53:11.967 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-13T23:53:11.973 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-13T23:53:12.121 INFO:teuthology.orchestra.run.smithi081.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-13T23:53:12.122 INFO:teuthology.orchestra.run.smithi081.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-13T23:53:12.122 INFO:teuthology.orchestra.run.smithi081.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-13T23:53:12.151 INFO:teuthology.orchestra.run.smithi081.stdout:done 2023-10-13T23:53:12.644 DEBUG:teuthology.orchestra.run.smithi081:> dpkg -s linux-image-generic 2023-10-13T23:53:12.666 INFO:teuthology.orchestra.run.smithi081.stdout:Package: linux-image-generic 2023-10-13T23:53:12.666 INFO:teuthology.orchestra.run.smithi081.stdout:Status: install ok installed 2023-10-13T23:53:12.667 INFO:teuthology.orchestra.run.smithi081.stdout:Priority: optional 2023-10-13T23:53:12.667 INFO:teuthology.orchestra.run.smithi081.stdout:Section: kernel 2023-10-13T23:53:12.667 INFO:teuthology.orchestra.run.smithi081.stdout:Installed-Size: 20 2023-10-13T23:53:12.667 INFO:teuthology.orchestra.run.smithi081.stdout:Maintainer: Ubuntu Kernel Team 2023-10-13T23:53:12.668 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: amd64 2023-10-13T23:53:12.668 INFO:teuthology.orchestra.run.smithi081.stdout:Source: linux-meta 2023-10-13T23:53:12.668 INFO:teuthology.orchestra.run.smithi081.stdout:Version: 5.15.0.86.83 2023-10-13T23:53:12.668 INFO:teuthology.orchestra.run.smithi081.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-13T23:53:12.669 INFO:teuthology.orchestra.run.smithi081.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-13T23:53:12.669 INFO:teuthology.orchestra.run.smithi081.stdout:Recommends: thermald 2023-10-13T23:53:12.669 INFO:teuthology.orchestra.run.smithi081.stdout:Description: Generic Linux kernel image 2023-10-13T23:53:12.669 INFO:teuthology.orchestra.run.smithi081.stdout: This package will always depend on the latest generic kernel image 2023-10-13T23:53:12.669 INFO:teuthology.orchestra.run.smithi081.stdout: available. 2023-10-13T23:53:12.670 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-86-generic 2023-10-13T23:53:12.670 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-13T23:53:12.670 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-13T23:53:12.670 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-13T23:53:12.671 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2023-10-13T23:53:12.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get clean 2023-10-13T23:53:12.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2023-10-13T23:53:12.967 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-13T23:53:12.968 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:53:12.981 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-13T23:53:13.001 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-13T23:53:14.175 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-13T23:53:14.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-13T23:53:14.242 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-13T23:53:14.405 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2023-10-13T23:53:14.405 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2023-10-13T23:53:14.547 INFO:teuthology.orchestra.run.smithi081.stdout:linux-image-generic is already the newest version (5.15.0.86.83). 2023-10-13T23:53:14.548 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:53:14.548 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:53:14.548 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:53:14.549 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:53:14.549 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-13T23:53:14.549 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-13T23:53:14.550 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:53:14.582 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 0 newly installed, 0 to remove and 268 not upgraded. 2023-10-13T23:53:14.584 DEBUG:teuthology.orchestra.run.smithi081:> dpkg -s linux-image-generic 2023-10-13T23:53:14.644 INFO:teuthology.orchestra.run.smithi081.stdout:Package: linux-image-generic 2023-10-13T23:53:14.644 INFO:teuthology.orchestra.run.smithi081.stdout:Status: install ok installed 2023-10-13T23:53:14.644 INFO:teuthology.orchestra.run.smithi081.stdout:Priority: optional 2023-10-13T23:53:14.645 INFO:teuthology.orchestra.run.smithi081.stdout:Section: kernel 2023-10-13T23:53:14.645 INFO:teuthology.orchestra.run.smithi081.stdout:Installed-Size: 20 2023-10-13T23:53:14.645 INFO:teuthology.orchestra.run.smithi081.stdout:Maintainer: Ubuntu Kernel Team 2023-10-13T23:53:14.645 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: amd64 2023-10-13T23:53:14.645 INFO:teuthology.orchestra.run.smithi081.stdout:Source: linux-meta 2023-10-13T23:53:14.646 INFO:teuthology.orchestra.run.smithi081.stdout:Version: 5.15.0.86.83 2023-10-13T23:53:14.646 INFO:teuthology.orchestra.run.smithi081.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-13T23:53:14.646 INFO:teuthology.orchestra.run.smithi081.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-13T23:53:14.646 INFO:teuthology.orchestra.run.smithi081.stdout:Recommends: thermald 2023-10-13T23:53:14.646 INFO:teuthology.orchestra.run.smithi081.stdout:Description: Generic Linux kernel image 2023-10-13T23:53:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: This package will always depend on the latest generic kernel image 2023-10-13T23:53:14.647 INFO:teuthology.orchestra.run.smithi081.stdout: available. 2023-10-13T23:53:14.647 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2023-10-13T23:53:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.vpuUJLMen0 2023-10-13T23:53:14.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vpuUJLMen0 2023-10-13T23:53:14.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.vpuUJLMen0 2023-10-13T23:53:14.877 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.vpuUJLMen0 is 10KB 2023-10-13T23:53:14.890 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.vpuUJLMen0 2023-10-13T23:53:14.896 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-13T23:53:14.948 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-13T23:53:14.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-13T23:53:14.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-13T23:53:15.047 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-86-generic 2023-10-13T23:53:15.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo update-grub 2023-10-13T23:53:16.261 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub' 2023-10-13T23:53:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-13T23:53:16.264 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2023-10-13T23:53:16.526 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:53:16.533 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:53:16.821 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:53:16.827 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-13T23:53:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-13T23:53:16.877 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-13T23:53:17.011 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-13T23:53:17.011 INFO:teuthology.orchestra.run.smithi081.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-13T23:53:17.012 INFO:teuthology.orchestra.run.smithi081.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-13T23:53:17.024 INFO:teuthology.orchestra.run.smithi081.stderr:done 2023-10-13T23:53:17.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2023-10-13T23:53:47.067 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-13T23:53:47.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-13T23:53:47.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:54:05.562 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-10-13T23:54:14.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-13T23:54:14.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:54:17.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-10-13T23:54:29.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-13T23:54:29.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:54:48.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-10-13T23:55:03.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-13T23:55:03.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:55:03.205 DEBUG:teuthology.orchestra.run.smithi081:> true 2023-10-13T23:55:03.757 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-13T23:55:03.757 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-86-generic"... 2023-10-13T23:55:03.757 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2023-10-13T23:55:03.804 INFO:teuthology.orchestra.run.smithi081.stdout:5.15.0-86-generic 2023-10-13T23:55:03.805 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-86-generic vs 5.15.0-86-generic 2023-10-13T23:55:03.805 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-13T23:55:03.805 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-13T23:55:04.806 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-13T23:55:04.807 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-13T23:55:04.882 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2023-10-13T23:55:04.914 DEBUG:teuthology.parallel:result is None 2023-10-13T23:55:04.915 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-13T23:55:04.922 INFO:teuthology.task.internal:Creating test directory... 2023-10-13T23:55:04.923 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-13T23:55:04.928 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-13T23:55:04.963 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-13T23:55:04.986 INFO:teuthology.task.internal:Creating archive directory... 2023-10-13T23:55:04.987 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-13T23:55:05.008 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-13T23:55:05.018 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-13T23:55:05.018 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-13T23:55:05.067 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-13T23:55:05.077 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-13T23:55:05.078 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-13T23:55:05.088 INFO:teuthology.task.internal:Configuring sudo... 2023-10-13T23:55:05.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-13T23:55:05.132 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-13T23:55:05.159 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-13T23:55:05.159 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-13T23:55:05.181 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-13T23:55:05.229 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-13T23:55:05.278 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-13T23:55:05.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-13T23:55:05.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2023-10-13T23:55:05.451 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-13T23:55:05.461 INFO:teuthology.task.internal:Starting timer... 2023-10-13T23:55:05.461 INFO:teuthology.run_tasks:Running task pcp... 2023-10-13T23:55:05.517 INFO:teuthology.run_tasks:Running task selinux... 2023-10-13T23:55:05.527 DEBUG:teuthology.task.selinux:Excluding smithi081: OS 'ubuntu' does not support SELinux 2023-10-13T23:55:05.527 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-13T23:55:05.528 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-13T23:55:05.528 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-13T23:55:05.528 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-13T23:55:05.538 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-13T23:55:05.538 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-13T23:55:05.556 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-13T23:55:05.557 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-14T00:02:51.607 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com')] 2023-10-14T00:02:51.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-14T00:02:51.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T00:02:51.690 DEBUG:teuthology.orchestra.run.smithi081:> true 2023-10-14T00:02:51.796 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-10-14T00:02:51.797 INFO:teuthology.run_tasks:Running task clock... 2023-10-14T00:02:51.830 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-14T00:02:51.832 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-14T00:02:51.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T00:02:51.871 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-14T00:02:51.872 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Command line: ntpd -gq 2023-10-14T00:02:51.873 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: ---------------------------------------------------- 2023-10-14T00:02:51.873 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: ntp-4 is maintained by Network Time Foundation, 2023-10-14T00:02:51.873 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-14T00:02:51.874 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: corporation. Support and training for ntp-4 are 2023-10-14T00:02:51.874 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: available at https://www.nwtime.org/support 2023-10-14T00:02:51.875 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: ---------------------------------------------------- 2023-10-14T00:02:51.875 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: proto: precision = 0.045 usec (-24) 2023-10-14T00:02:51.875 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: basedate set to 2022-02-04 2023-10-14T00:02:51.876 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: gps base set to 2022-02-06 (week 2196) 2023-10-14T00:02:51.876 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-14T00:02:51.877 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: restrict ::: KOD does nothing without LIMITED. 2023-10-14T00:02:51.877 INFO:teuthology.orchestra.run.smithi081.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-14T00:02:51.878 INFO:teuthology.orchestra.run.smithi081.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-14T00:02:51.881 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listen and drop on 0 v6wildcard [::]:123 2023-10-14T00:02:51.881 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-14T00:02:51.881 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listen normally on 2 lo 127.0.0.1:123 2023-10-14T00:02:51.882 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listen normally on 3 enp3s0f1 172.21.15.81:123 2023-10-14T00:02:51.882 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listen normally on 4 lo [::1]:123 2023-10-14T00:02:51.882 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d753%5]:123 2023-10-14T00:02:51.882 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:51 ntpd[28092]: Listening on routing socket on fd #22 for interface updates 2023-10-14T00:02:52.871 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:52 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:52.872 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:52 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:52.872 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:52 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:52.872 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:52 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:52.872 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:52 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:52.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:52 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:53.871 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:53 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:53.871 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:53 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:54.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:54.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:54.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:54.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:54.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:54.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:54.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:54.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:54.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:54.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:54.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:54.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:54 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:56.872 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:56.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:56.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:56.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:56.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:56.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:56.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:56.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:56.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:56.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:56.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:56.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:56.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:56.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:56.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:56.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:56.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:56.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:56 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:58.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:58.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:58.873 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:58.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:58.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:58.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:58.874 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:58.877 INFO:teuthology.orchestra.run.smithi081.stdout:14 Oct 00:02:58 ntpd[28092]: ntpd: time slew -0.000495 s 2023-10-14T00:02:58.877 INFO:teuthology.orchestra.run.smithi081.stdout:ntpd: time slew -0.000495s 2023-10-14T00:02:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-14T00:02:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:14 Oct 00:02:58 ntpd[28092]: can't open /var/log/ntpstats/loopstats.20231014: Permission denied 2023-10-14T00:02:59.015 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2023-10-14T00:02:59.015 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2023-10-14T00:02:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:59.016 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:59.017 INFO:teuthology.run_tasks:Running task install... 2023-10-14T00:02:59.032 DEBUG:teuthology.task.install:project ceph 2023-10-14T00:02:59.032 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}} 2023-10-14T00:02:59.032 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'} 2023-10-14T00:02:59.033 INFO:teuthology.task.install:Using flavor: default 2023-10-14T00:02:59.039 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-14T00:02:59.039 INFO:teuthology.task.install:extra packages: [] 2023-10-14T00:02:59.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-key list | grep Ceph 2023-10-14T00:02:59.116 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-14T00:02:59.145 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-14T00:02:59.145 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph.com (release key) 2023-10-14T00:02:59.145 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-14T00:02:59.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:02:59.391 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default/ 2023-10-14T00:02:59.392 INFO:teuthology.task.install.deb:Package version is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:02:59.574 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:02:59.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-14T00:02:59.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2023-10-14T00:02:59.882 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-14T00:02:59.882 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-14T00:02:59.954 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-14T00:02:59.982 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy InRelease 2023-10-14T00:03:00.036 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-14T00:03:00.050 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-14T00:03:00.117 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-14T00:03:00.184 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-10-14T00:03:00.258 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2023-10-14T00:03:00.335 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 28.5 kB in 1s (50.4 kB/s) 2023-10-14T00:03:01.205 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-14T00:03:01.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6724-g01fff6a7-1jammy cephadm=18.0.0-6724-g01fff6a7-1jammy ceph-mds=18.0.0-6724-g01fff6a7-1jammy ceph-mgr=18.0.0-6724-g01fff6a7-1jammy ceph-common=18.0.0-6724-g01fff6a7-1jammy ceph-fuse=18.0.0-6724-g01fff6a7-1jammy ceph-test=18.0.0-6724-g01fff6a7-1jammy ceph-volume=18.0.0-6724-g01fff6a7-1jammy radosgw=18.0.0-6724-g01fff6a7-1jammy python3-rados=18.0.0-6724-g01fff6a7-1jammy python3-rgw=18.0.0-6724-g01fff6a7-1jammy python3-cephfs=18.0.0-6724-g01fff6a7-1jammy python3-rbd=18.0.0-6724-g01fff6a7-1jammy libcephfs2=18.0.0-6724-g01fff6a7-1jammy libcephfs-dev=18.0.0-6724-g01fff6a7-1jammy librados2=18.0.0-6724-g01fff6a7-1jammy librbd1=18.0.0-6724-g01fff6a7-1jammy rbd-fuse=18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:03:01.280 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-14T00:03:01.465 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2023-10-14T00:03:01.466 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2023-10-14T00:03:01.626 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2023-10-14T00:03:01.627 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-14T00:03:01.627 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-14T00:03:01.627 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-14T00:03:01.627 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-14T00:03:01.629 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-14T00:03:01.629 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-14T00:03:01.629 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2023-10-14T00:03:01.629 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-14T00:03:01.630 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-14T00:03:01.630 INFO:teuthology.orchestra.run.smithi081.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-14T00:03:01.630 INFO:teuthology.orchestra.run.smithi081.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-14T00:03:01.630 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-14T00:03:01.630 INFO:teuthology.orchestra.run.smithi081.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-14T00:03:01.631 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-14T00:03:01.631 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-14T00:03:01.631 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-14T00:03:01.631 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-14T00:03:01.631 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-14T00:03:01.632 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-14T00:03:01.632 INFO:teuthology.orchestra.run.smithi081.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-14T00:03:01.632 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-14T00:03:01.632 INFO:teuthology.orchestra.run.smithi081.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-14T00:03:01.632 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-14T00:03:01.634 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-14T00:03:01.634 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-14T00:03:01.634 INFO:teuthology.orchestra.run.smithi081.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-14T00:03:01.634 INFO:teuthology.orchestra.run.smithi081.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-14T00:03:01.634 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-14T00:03:01.635 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-14T00:03:01.635 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2023-10-14T00:03:01.635 INFO:teuthology.orchestra.run.smithi081.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-14T00:03:01.636 INFO:teuthology.orchestra.run.smithi081.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-14T00:03:01.636 INFO:teuthology.orchestra.run.smithi081.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-14T00:03:01.636 INFO:teuthology.orchestra.run.smithi081.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-14T00:03:01.636 INFO:teuthology.orchestra.run.smithi081.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-14T00:03:01.636 INFO:teuthology.orchestra.run.smithi081.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-14T00:03:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: python3-watchdog 2023-10-14T00:03:01.637 INFO:teuthology.orchestra.run.smithi081.stdout:Recommended packages: 2023-10-14T00:03:01.637 INFO:teuthology.orchestra.run.smithi081.stdout: btrfs-tools 2023-10-14T00:03:01.723 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2023-10-14T00:03:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-14T00:03:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-14T00:03:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-14T00:03:01.724 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-14T00:03:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-14T00:03:01.725 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-14T00:03:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-14T00:03:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-14T00:03:01.726 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-14T00:03:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-14T00:03:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-14T00:03:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-14T00:03:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-14T00:03:01.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-14T00:03:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-14T00:03:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-14T00:03:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-14T00:03:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-14T00:03:01.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-14T00:03:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-14T00:03:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-14T00:03:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-14T00:03:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-14T00:03:01.729 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-14T00:03:01.730 INFO:teuthology.orchestra.run.smithi081.stdout: socat xmlstarlet 2023-10-14T00:03:02.081 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 109 newly installed, 0 to remove and 256 not upgraded. 2023-10-14T00:03:02.081 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 184 MB of archives. 2023-10-14T00:03:02.081 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-14T00:03:02.082 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6724-g01fff6a7-1jammy [3,450 kB] 2023-10-14T00:03:02.687 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-14T00:03:02.744 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-14T00:03:02.781 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-14T00:03:02.783 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-14T00:03:02.785 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6724-g01fff6a7-1jammy [3,130 kB] 2023-10-14T00:03:02.798 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-14T00:03:02.849 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-14T00:03:02.854 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-14T00:03:02.866 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-14T00:03:02.870 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-14T00:03:02.871 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-14T00:03:02.871 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-14T00:03:02.880 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-14T00:03:02.880 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-14T00:03:02.881 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-14T00:03:02.882 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-14T00:03:02.887 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-14T00:03:02.887 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-14T00:03:02.904 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-14T00:03:02.904 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-14T00:03:02.905 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-14T00:03:02.905 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-14T00:03:02.908 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-14T00:03:02.908 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-14T00:03:02.909 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-14T00:03:02.909 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-14T00:03:02.920 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-14T00:03:02.921 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-14T00:03:02.942 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-14T00:03:02.942 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-14T00:03:02.943 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-14T00:03:02.943 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-14T00:03:02.948 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-14T00:03:02.948 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-14T00:03:02.948 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-14T00:03:02.949 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-14T00:03:02.957 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-14T00:03:02.959 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-14T00:03:02.967 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-14T00:03:02.969 INFO:teuthology.orchestra.run.smithi081.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-14T00:03:02.969 INFO:teuthology.orchestra.run.smithi081.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-14T00:03:02.972 INFO:teuthology.orchestra.run.smithi081.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-14T00:03:02.972 INFO:teuthology.orchestra.run.smithi081.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-14T00:03:02.975 INFO:teuthology.orchestra.run.smithi081.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-14T00:03:03.027 INFO:teuthology.orchestra.run.smithi081.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-14T00:03:03.027 INFO:teuthology.orchestra.run.smithi081.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-14T00:03:03.029 INFO:teuthology.orchestra.run.smithi081.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-14T00:03:03.029 INFO:teuthology.orchestra.run.smithi081.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-14T00:03:03.039 INFO:teuthology.orchestra.run.smithi081.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-14T00:03:03.040 INFO:teuthology.orchestra.run.smithi081.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-14T00:03:03.058 INFO:teuthology.orchestra.run.smithi081.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-14T00:03:03.059 INFO:teuthology.orchestra.run.smithi081.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-14T00:03:03.060 INFO:teuthology.orchestra.run.smithi081.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-14T00:03:03.075 INFO:teuthology.orchestra.run.smithi081.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-14T00:03:03.075 INFO:teuthology.orchestra.run.smithi081.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-14T00:03:03.076 INFO:teuthology.orchestra.run.smithi081.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-14T00:03:03.076 INFO:teuthology.orchestra.run.smithi081.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-14T00:03:03.077 INFO:teuthology.orchestra.run.smithi081.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-14T00:03:03.077 INFO:teuthology.orchestra.run.smithi081.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-14T00:03:03.085 INFO:teuthology.orchestra.run.smithi081.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-14T00:03:03.085 INFO:teuthology.orchestra.run.smithi081.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-14T00:03:03.089 INFO:teuthology.orchestra.run.smithi081.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-14T00:03:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-14T00:03:03.103 INFO:teuthology.orchestra.run.smithi081.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-14T00:03:03.104 INFO:teuthology.orchestra.run.smithi081.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-14T00:03:03.107 INFO:teuthology.orchestra.run.smithi081.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-14T00:03:03.109 INFO:teuthology.orchestra.run.smithi081.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-14T00:03:03.113 INFO:teuthology.orchestra.run.smithi081.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-14T00:03:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-14T00:03:03.119 INFO:teuthology.orchestra.run.smithi081.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-14T00:03:03.119 INFO:teuthology.orchestra.run.smithi081.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-14T00:03:03.120 INFO:teuthology.orchestra.run.smithi081.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-14T00:03:03.136 INFO:teuthology.orchestra.run.smithi081.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-14T00:03:03.136 INFO:teuthology.orchestra.run.smithi081.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-14T00:03:03.143 INFO:teuthology.orchestra.run.smithi081.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-14T00:03:03.143 INFO:teuthology.orchestra.run.smithi081.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-14T00:03:03.143 INFO:teuthology.orchestra.run.smithi081.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-14T00:03:03.145 INFO:teuthology.orchestra.run.smithi081.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-14T00:03:03.145 INFO:teuthology.orchestra.run.smithi081.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-14T00:03:03.223 INFO:teuthology.orchestra.run.smithi081.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6724-g01fff6a7-1jammy [688 kB] 2023-10-14T00:03:03.279 INFO:teuthology.orchestra.run.smithi081.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6724-g01fff6a7-1jammy [356 kB] 2023-10-14T00:03:03.303 INFO:teuthology.orchestra.run.smithi081.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6724-g01fff6a7-1jammy [32.8 kB] 2023-10-14T00:03:03.303 INFO:teuthology.orchestra.run.smithi081.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6724-g01fff6a7-1jammy [180 kB] 2023-10-14T00:03:03.321 INFO:teuthology.orchestra.run.smithi081.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6724-g01fff6a7-1jammy [62.4 kB] 2023-10-14T00:03:03.323 INFO:teuthology.orchestra.run.smithi081.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6724-g01fff6a7-1jammy [330 kB] 2023-10-14T00:03:03.346 INFO:teuthology.orchestra.run.smithi081.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6724-g01fff6a7-1jammy [6,222 kB] 2023-10-14T00:03:03.800 INFO:teuthology.orchestra.run.smithi081.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6724-g01fff6a7-1jammy [112 kB] 2023-10-14T00:03:03.802 INFO:teuthology.orchestra.run.smithi081.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6724-g01fff6a7-1jammy [439 kB] 2023-10-14T00:03:03.834 INFO:teuthology.orchestra.run.smithi081.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6724-g01fff6a7-1jammy [22.6 MB] 2023-10-14T00:03:06.530 INFO:teuthology.orchestra.run.smithi081.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6724-g01fff6a7-1jammy [6,853 kB] 2023-10-14T00:03:07.789 INFO:teuthology.orchestra.run.smithi081.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6724-g01fff6a7-1jammy [238 kB] 2023-10-14T00:03:07.834 INFO:teuthology.orchestra.run.smithi081.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6724-g01fff6a7-1jammy [128 kB] 2023-10-14T00:03:07.854 INFO:teuthology.orchestra.run.smithi081.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6724-g01fff6a7-1jammy [1,496 kB] 2023-10-14T00:03:08.200 INFO:teuthology.orchestra.run.smithi081.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6724-g01fff6a7-1jammy [9,863 kB] 2023-10-14T00:03:10.203 INFO:teuthology.orchestra.run.smithi081.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6724-g01fff6a7-1jammy [31.7 MB] 2023-10-14T00:03:12.544 INFO:teuthology.orchestra.run.smithi081.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6724-g01fff6a7-1jammy [14.0 kB] 2023-10-14T00:03:12.545 INFO:teuthology.orchestra.run.smithi081.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6724-g01fff6a7-1jammy [839 kB] 2023-10-14T00:03:12.567 INFO:teuthology.orchestra.run.smithi081.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6724-g01fff6a7-1jammy [2,146 kB] 2023-10-14T00:03:12.689 INFO:teuthology.orchestra.run.smithi081.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6724-g01fff6a7-1jammy [283 kB] 2023-10-14T00:03:12.694 INFO:teuthology.orchestra.run.smithi081.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6724-g01fff6a7-1jammy [138 kB] 2023-10-14T00:03:12.710 INFO:teuthology.orchestra.run.smithi081.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6724-g01fff6a7-1jammy [1,906 kB] 2023-10-14T00:03:12.821 INFO:teuthology.orchestra.run.smithi081.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6724-g01fff6a7-1jammy [8,625 kB] 2023-10-14T00:03:13.297 INFO:teuthology.orchestra.run.smithi081.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6724-g01fff6a7-1jammy [14.2 kB] 2023-10-14T00:03:13.297 INFO:teuthology.orchestra.run.smithi081.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6724-g01fff6a7-1jammy [47.8 MB] 2023-10-14T00:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6724-g01fff6a7-1jammy [120 kB] 2023-10-14T00:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6724-g01fff6a7-1jammy [38.3 kB] 2023-10-14T00:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6724-g01fff6a7-1jammy [12.3 MB] 2023-10-14T00:03:16.388 INFO:teuthology.orchestra.run.smithi081.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6724-g01fff6a7-1jammy [91.4 kB] 2023-10-14T00:03:16.939 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 184 MB in 15s (12.5 MB/s) 2023-10-14T00:03:17.077 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-14T00:03:17.173 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2023-10-14T00:03:17.178 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-14T00:03:17.259 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-14T00:03:17.595 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-14T00:03:17.621 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-14T00:03:17.661 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-14T00:03:18.006 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-14T00:03:18.033 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-14T00:03:18.073 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-14T00:03:18.409 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-14T00:03:18.435 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-14T00:03:18.476 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-14T00:03:18.890 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-14T00:03:18.914 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-14T00:03:18.963 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:03:19.300 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-14T00:03:19.326 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-14T00:03:19.358 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:03:19.702 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-14T00:03:19.728 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-14T00:03:19.769 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:03:20.114 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-14T00:03:20.140 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-14T00:03:20.180 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-14T00:03:20.534 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librados2. 2023-10-14T00:03:20.560 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../008-librados2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:20.600 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librados2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:21.028 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librbd1. 2023-10-14T00:03:21.054 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../009-librbd1_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:21.094 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librbd1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:21.506 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libcephfs2. 2023-10-14T00:03:21.532 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:21.572 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libcephfs2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:22.083 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rados. 2023-10-14T00:03:22.109 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:22.150 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rados (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:22.478 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-14T00:03:22.504 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:22.544 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-ceph-argparse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:22.805 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cephfs. 2023-10-14T00:03:22.820 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:22.855 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cephfs (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:23.141 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-14T00:03:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:23.199 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:23.527 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-14T00:03:23.553 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-14T00:03:23.594 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-14T00:03:23.905 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-prettytable. 2023-10-14T00:03:23.931 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-14T00:03:23.972 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-14T00:03:24.216 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rbd. 2023-10-14T00:03:24.231 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:24.266 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rbd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:24.569 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-14T00:03:24.596 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-14T00:03:24.636 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-14T00:03:24.963 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librgw2. 2023-10-14T00:03:24.989 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../019-librgw2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:25.030 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librgw2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:25.557 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rgw. 2023-10-14T00:03:25.573 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:25.615 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:25.935 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-14T00:03:25.961 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-14T00:03:25.993 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-14T00:03:26.305 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libradosstriper1. 2023-10-14T00:03:26.331 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:26.371 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libradosstriper1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:26.791 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-common. 2023-10-14T00:03:26.817 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:26.859 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:28.131 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-base. 2023-10-14T00:03:28.157 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:28.206 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-base (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:28.910 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-14T00:03:28.936 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-14T00:03:28.976 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-14T00:03:29.279 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cheroot. 2023-10-14T00:03:29.305 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-14T00:03:29.346 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-14T00:03:29.707 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-14T00:03:29.733 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-14T00:03:29.773 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-14T00:03:30.060 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-14T00:03:30.086 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-14T00:03:30.118 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-14T00:03:30.404 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-14T00:03:30.430 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-14T00:03:30.471 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-14T00:03:30.774 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-tempora. 2023-10-14T00:03:30.800 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-14T00:03:30.840 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-14T00:03:31.151 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-portend. 2023-10-14T00:03:31.167 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-14T00:03:31.210 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-14T00:03:31.463 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-14T00:03:31.489 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-14T00:03:31.529 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-14T00:03:31.832 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-14T00:03:31.858 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-14T00:03:31.898 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-14T00:03:32.219 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-natsort. 2023-10-14T00:03:32.234 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-14T00:03:32.269 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-14T00:03:32.530 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-logutils. 2023-10-14T00:03:32.556 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-14T00:03:32.588 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-14T00:03:33.033 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-14T00:03:33.059 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-14T00:03:33.100 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-14T00:03:33.386 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-mako. 2023-10-14T00:03:33.413 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-14T00:03:33.452 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-14T00:03:33.806 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-14T00:03:33.833 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-14T00:03:33.872 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-14T00:03:34.133 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-14T00:03:34.148 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-14T00:03:34.183 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-14T00:03:34.486 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-webob. 2023-10-14T00:03:34.513 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-14T00:03:34.553 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-14T00:03:35.481 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-waitress. 2023-10-14T00:03:35.509 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-14T00:03:35.543 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-14T00:03:35.826 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-tempita. 2023-10-14T00:03:35.852 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-14T00:03:35.892 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-14T00:03:36.195 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-paste. 2023-10-14T00:03:36.221 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-14T00:03:36.262 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-14T00:03:36.581 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-14T00:03:36.597 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-14T00:03:36.640 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-14T00:03:36.951 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-14T00:03:36.977 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-14T00:03:37.018 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-14T00:03:37.304 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-webtest. 2023-10-14T00:03:37.330 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-14T00:03:37.387 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-14T00:03:37.673 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pecan. 2023-10-14T00:03:37.701 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-14T00:03:37.731 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-14T00:03:38.165 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-certifi. 2023-10-14T00:03:38.191 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-14T00:03:38.263 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-14T00:03:38.583 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-idna. 2023-10-14T00:03:38.609 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-14T00:03:38.649 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-14T00:03:38.927 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-urllib3. 2023-10-14T00:03:38.954 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-14T00:03:38.985 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-14T00:03:39.281 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-requests. 2023-10-14T00:03:39.307 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-14T00:03:39.339 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-14T00:03:39.600 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-14T00:03:39.626 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-14T00:03:39.666 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-14T00:03:39.936 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-14T00:03:39.963 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:40.003 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:40.333 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-14T00:03:40.348 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:40.383 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:40.669 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-openssl. 2023-10-14T00:03:40.695 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-14T00:03:40.735 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-14T00:03:41.080 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr. 2023-10-14T00:03:41.107 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:41.147 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:41.567 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mon. 2023-10-14T00:03:41.593 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:41.633 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mon (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:42.562 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-osd. 2023-10-14T00:03:42.588 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:42.628 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-osd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:44.457 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph. 2023-10-14T00:03:44.472 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../059-ceph_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:44.506 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:44.801 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-fuse. 2023-10-14T00:03:44.827 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:44.869 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:45.263 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mds. 2023-10-14T00:03:45.290 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:45.329 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mds (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:45.708 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cephadm. 2023-10-14T00:03:45.734 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../062-cephadm_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:45.774 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:46.110 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-14T00:03:46.137 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-14T00:03:46.177 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-14T00:03:46.488 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-14T00:03:46.503 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-14T00:03:46.538 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-14T00:03:47.468 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-babel. 2023-10-14T00:03:47.483 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-14T00:03:47.527 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-14T00:03:47.838 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jinja2. 2023-10-14T00:03:47.865 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-14T00:03:47.905 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-14T00:03:48.233 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-14T00:03:48.259 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:48.299 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:48.820 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-14T00:03:48.846 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-14T00:03:48.920 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-14T00:03:49.223 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-14T00:03:49.249 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-14T00:03:49.289 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-14T00:03:49.567 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-routes. 2023-10-14T00:03:49.594 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-14T00:03:49.634 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-14T00:03:49.987 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-14T00:03:50.014 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:50.054 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:50.474 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-14T00:03:50.489 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-14T00:03:50.532 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-14T00:03:51.019 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-joblib. 2023-10-14T00:03:51.046 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-14T00:03:51.085 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-14T00:03:51.448 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-14T00:03:51.475 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-14T00:03:51.514 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-14T00:03:51.784 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-sklearn. 2023-10-14T00:03:51.811 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-14T00:03:51.842 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-14T00:03:52.534 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-14T00:03:52.562 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:52.601 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cachetools. 2023-10-14T00:03:53.557 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-14T00:03:53.587 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-14T00:03:53.874 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-14T00:03:53.893 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-14T00:03:53.932 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-14T00:03:54.244 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-14T00:03:54.270 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-14T00:03:54.310 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-14T00:03:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rsa. 2023-10-14T00:03:54.657 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-14T00:03:54.696 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-14T00:03:54.992 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-google-auth. 2023-10-14T00:03:55.018 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-14T00:03:55.058 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-14T00:03:55.395 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-14T00:03:55.422 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-14T00:03:55.461 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-14T00:03:55.824 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-websocket. 2023-10-14T00:03:55.839 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-14T00:03:55.874 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-14T00:03:56.153 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-14T00:03:56.179 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-14T00:03:56.219 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-14T00:03:57.008 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-14T00:03:57.024 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:57.058 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:57.361 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-14T00:03:57.388 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-14T00:03:57.427 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-14T00:03:57.756 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-14T00:03:57.783 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-14T00:03:57.822 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-14T00:03:58.033 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package jq. 2023-10-14T00:03:58.060 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-14T00:03:58.091 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-14T00:03:58.344 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package socat. 2023-10-14T00:03:58.360 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-14T00:03:58.402 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-14T00:03:58.714 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package xmlstarlet. 2023-10-14T00:03:58.741 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-14T00:03:58.781 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-14T00:03:59.319 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-test. 2023-10-14T00:03:59.346 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:59.385 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-test (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:01.656 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-volume. 2023-10-14T00:04:01.683 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:04:01.722 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-volume (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.010 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-14T00:04:02.037 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:04:02.068 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libcephfs-dev (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.429 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package nvme-cli. 2023-10-14T00:04:02.456 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-14T00:04:02.496 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-14T00:04:02.834 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-14T00:04:02.850 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-14T00:04:02.893 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-14T00:04:03.321 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-14T00:04:03.348 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-14T00:04:03.387 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-14T00:04:03.690 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pastescript. 2023-10-14T00:04:03.717 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-14T00:04:03.757 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-14T00:04:04.094 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pluggy. 2023-10-14T00:04:04.121 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-14T00:04:04.160 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-14T00:04:04.505 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-psutil. 2023-10-14T00:04:04.532 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-14T00:04:04.572 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-14T00:04:04.908 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-py. 2023-10-14T00:04:04.935 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-14T00:04:04.974 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-14T00:04:05.286 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pygments. 2023-10-14T00:04:05.314 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-14T00:04:05.344 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-14T00:04:05.807 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-14T00:04:05.834 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-14T00:04:05.873 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-14T00:04:06.176 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-toml. 2023-10-14T00:04:06.204 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-14T00:04:06.243 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-14T00:04:06.546 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pytest. 2023-10-14T00:04:06.573 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-14T00:04:06.612 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-14T00:04:06.933 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-simplejson. 2023-10-14T00:04:06.949 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-14T00:04:06.982 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-14T00:04:07.252 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-14T00:04:07.267 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-14T00:04:07.301 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-14T00:04:07.981 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package radosgw. 2023-10-14T00:04:08.008 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../107-radosgw_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:04:08.047 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking radosgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:08.800 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package rbd-fuse. 2023-10-14T00:04:08.828 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:04:08.867 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking rbd-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:09.153 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-14T00:04:09.397 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-14T00:04:09.509 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-14T00:04:09.768 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-14T00:04:10.085 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-14T00:04:10.524 INFO:teuthology.orchestra.run.smithi081.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-14T00:04:10.616 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:11.126 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user cephadm....done 2023-10-14T00:04:11.290 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-14T00:04:11.566 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-14T00:04:11.817 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-14T00:04:11.943 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-14T00:04:12.202 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-14T00:04:12.454 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-14T00:04:12.555 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-14T00:04:12.840 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-14T00:04:13.192 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-14T00:04:13.443 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-14T00:04:13.694 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-14T00:04:13.937 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-14T00:04:14.204 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-14T00:04:14.448 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-ceph-argparse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:15.115 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-14T00:04:15.241 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-14T00:04:15.492 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-14T00:04:15.593 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-14T00:04:15.944 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-14T00:04:16.079 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-14T00:04:16.330 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-14T00:04:16.581 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-14T00:04:16.891 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-14T00:04:17.576 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-14T00:04:17.827 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-14T00:04:17.936 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-14T00:04:18.062 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-14T00:04:18.372 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-14T00:04:18.740 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-14T00:04:18.991 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-14T00:04:19.375 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-14T00:04:19.652 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-14T00:04:19.995 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-14T00:04:20.237 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-14T00:04:20.355 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:20.649 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-idna (3.3-1) ... 2023-10-14T00:04:20.966 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-14T00:04:22.157 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:04:22.292 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-14T00:04:22.543 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-14T00:04:22.669 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-14T00:04:22.795 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-14T00:04:23.071 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-14T00:04:23.322 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-14T00:04:23.582 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:04:23.725 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-14T00:04:23.834 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-14T00:04:24.119 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-14T00:04:24.378 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-14T00:04:24.629 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-14T00:04:24.864 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-14T00:04:25.098 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-14T00:04:25.357 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-14T00:04:25.484 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-14T00:04:25.760 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-14T00:04:25.869 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-14T00:04:26.170 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-14T00:04:26.455 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-14T00:04:26.705 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-14T00:04:26.886 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-14T00:04:27.133 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-14T00:04:27.242 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-14T00:04:27.627 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-14T00:04:27.878 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-14T00:04:28.113 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:04:28.214 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-14T00:04:28.515 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-14T00:04:28.816 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-14T00:04:29.092 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-14T00:04:29.235 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-14T00:04:29.494 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-14T00:04:29.762 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-14T00:04:29.996 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-14T00:04:30.565 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-14T00:04:30.858 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-14T00:04:31.201 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-14T00:04:31.327 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librados2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:31.470 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librgw2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:31.571 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:31.672 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libcephfs2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:31.781 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libradosstriper1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:31.985 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-14T00:04:32.253 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librbd1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:32.379 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-modules-core (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:32.505 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:32.717 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-14T00:04:32.717 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-14T00:04:33.231 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libcephfs-dev (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:33.358 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rados (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:33.484 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-14T00:04:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rbd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:34.814 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up rbd-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:34.949 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:35.067 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cephfs (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:35.168 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:35.694 INFO:teuthology.orchestra.run.smithi081.stdout:Adding group ceph....done 2023-10-14T00:04:36.021 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user ceph....done 2023-10-14T00:04:36.198 INFO:teuthology.orchestra.run.smithi081.stdout:Setting system user ceph properties....done 2023-10-14T00:04:36.212 INFO:teuthology.orchestra.run.smithi081.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-14T00:04:36.351 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-14T00:04:36.693 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-14T00:04:37.186 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-test (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:37.312 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up radosgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:37.792 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-14T00:04:37.792 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-14T00:04:38.239 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-base (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:38.596 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-14T00:04:39.082 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mds (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:39.305 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-14T00:04:39.305 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-14T00:04:39.784 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:39.984 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-14T00:04:39.985 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-14T00:04:40.503 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-osd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:40.817 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-14T00:04:40.817 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-14T00:04:41.362 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:41.463 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:41.606 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mon (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:41.825 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-14T00:04:41.826 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-14T00:04:42.366 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:42.517 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:42.618 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-dashboard (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:42.744 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-volume (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:43.080 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-14T00:04:44.198 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-14T00:04:44.652 INFO:teuthology.orchestra.run.smithi081.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-14T00:04:44.655 DEBUG:teuthology.parallel:result is None 2023-10-14T00:04:44.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:04:44.873 DEBUG:teuthology.orchestra.run.smithi081:> dpkg-query -W -f '${Version}' ceph 2023-10-14T00:04:44.892 INFO:teuthology.orchestra.run.smithi081.stdout:18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:04:44.893 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:04:44.893 INFO:teuthology.task.install:The correct ceph version 18.0.0-6724-g01fff6a7-1jammy is installed. 2023-10-14T00:04:44.894 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-14T00:04:44.894 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:44.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-14T00:04:44.950 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-14T00:04:44.950 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:44.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2023-10-14T00:04:45.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-14T00:04:45.061 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-14T00:04:45.061 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:45.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-14T00:04:45.116 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-14T00:04:45.168 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-14T00:04:45.169 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:45.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2023-10-14T00:04:45.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-14T00:04:45.281 INFO:teuthology.run_tasks:Running task ceph... 2023-10-14T00:04:45.433 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-14T00:04:45.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2023-10-14T00:04:45.447 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-14T00:04:45.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-14T00:04:45.500 INFO:tasks.ceph:Creating extra log directories... 2023-10-14T00:04:45.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-14T00:04:45.578 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-14T00:04:45.578 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-14T00:04:45.579 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279', 'branch': 'wip-dis-testing', 'description': 'rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7427279', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': '735f68d7bf91d643813a8420fc2b11441f4bd8c4'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2449, 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f', 'sleep_before_teardown': 0, 'subset': '49/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': '735f68d7bf91d643813a8420fc2b11441f4bd8c4', 'targets': {'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGpmrSRWtaEDZQ9e8lI3BFxZn+4PtOyoHSrzMszzz2knv/8eHKNBninwN6j44sm9RCFUamsrbYa94st1WdqRN4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-13_23:35:38', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-14T00:04:45.579 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-14T00:04:45.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2023-10-14T00:04:45.661 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:45.662 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2023-10-14T00:04:45.710 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-14T00:04:45.710 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2023-10-14T00:04:45.758 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-14T00:04:45.758 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:45.758 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 885 Links: 1 2023-10-14T00:04:45.758 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:45.759 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2023-10-13 23:59:32.330632578 +0000 2023-10-14T00:04:45.759 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2023-10-13 23:59:31.990635614 +0000 2023-10-14T00:04:45.759 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2023-10-13 23:59:31.990635614 +0000 2023-10-14T00:04:45.759 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2023-10-14T00:04:45.760 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-14T00:04:45.812 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2023-10-14T00:04:45.812 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2023-10-14T00:04:45.813 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000437104 s, 1.2 MB/s 2023-10-14T00:04:45.814 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-14T00:04:45.863 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2023-10-14T00:04:45.913 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-14T00:04:45.914 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 906 Links: 1 2023-10-14T00:04:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:45.915 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2023-10-13 23:59:32.930627219 +0000 2023-10-14T00:04:45.915 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2023-10-13 23:59:32.558630542 +0000 2023-10-14T00:04:45.915 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2023-10-13 23:59:32.558630542 +0000 2023-10-14T00:04:45.916 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2023-10-14T00:04:45.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-14T00:04:45.971 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2023-10-14T00:04:45.972 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2023-10-14T00:04:45.972 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000366857 s, 1.4 MB/s 2023-10-14T00:04:45.973 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-14T00:04:46.022 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2023-10-14T00:04:46.069 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-14T00:04:46.070 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:46.070 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-10-14T00:04:46.070 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:46.070 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2023-10-13 23:59:33.166625111 +0000 2023-10-14T00:04:46.070 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2023-10-13 23:59:33.162625146 +0000 2023-10-14T00:04:46.071 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2023-10-13 23:59:33.162625146 +0000 2023-10-14T00:04:46.071 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2023-10-14T00:04:46.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-14T00:04:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2023-10-14T00:04:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2023-10-14T00:04:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000301358 s, 1.7 MB/s 2023-10-14T00:04:46.124 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-14T00:04:46.175 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2023-10-14T00:04:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-14T00:04:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:46.222 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 935 Links: 1 2023-10-14T00:04:46.222 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:46.222 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2023-10-13 23:59:34.086616892 +0000 2023-10-14T00:04:46.223 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2023-10-13 23:59:33.714620215 +0000 2023-10-14T00:04:46.223 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2023-10-13 23:59:33.714620215 +0000 2023-10-14T00:04:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2023-10-14T00:04:46.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-14T00:04:46.275 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2023-10-14T00:04:46.275 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2023-10-14T00:04:46.275 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00028165 s, 1.8 MB/s 2023-10-14T00:04:46.277 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-14T00:04:46.327 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:46.327 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-14T00:04:46.327 INFO:tasks.ceph:Generating config... 2023-10-14T00:04:46.329 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-14T00:04:46.329 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-14T00:04:46.329 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-14T00:04:46.330 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-14T00:04:46.330 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-14T00:04:46.330 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-14T00:04:46.331 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-14T00:04:46.331 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-14T00:04:46.331 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-14T00:04:46.331 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-14T00:04:46.332 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-14T00:04:46.332 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-14T00:04:46.332 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-14T00:04:46.333 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-14T00:04:46.333 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-14T00:04:46.333 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-14T00:04:46.333 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-14T00:04:46.334 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-14T00:04:46.334 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-14T00:04:46.334 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-14T00:04:46.335 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-14T00:04:46.335 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-14T00:04:46.335 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-14T00:04:46.335 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-14T00:04:46.336 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-14T00:04:46.336 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-14T00:04:46.336 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-14T00:04:46.337 INFO:tasks.ceph:Setting up mon.a... 2023-10-14T00:04:46.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-14T00:04:46.400 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.keyring 2023-10-14T00:04:46.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-14T00:04:46.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T00:04:46.540 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.81')] 2023-10-14T00:04:46.541 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.81', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-14T00:04:46.542 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:46.542 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T00:04:46.585 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.81 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:46.663 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:46.664 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: generated fsid e624b2c7-92dd-4594-9e39-e4d11116e328 2023-10-14T00:04:46.664 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = pacific 2023-10-14T00:04:46.664 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 0 2023-10-14T00:04:46.664 INFO:teuthology.orchestra.run.smithi081.stdout:fsid e624b2c7-92dd-4594-9e39-e4d11116e328 2023-10-14T00:04:46.665 INFO:teuthology.orchestra.run.smithi081.stdout:last_changed 2023-10-14T00:04:46.664991+0000 2023-10-14T00:04:46.665 INFO:teuthology.orchestra.run.smithi081.stdout:created 2023-10-14T00:04:46.664991+0000 2023-10-14T00:04:46.665 INFO:teuthology.orchestra.run.smithi081.stdout:min_mon_release 16 (pacific) 2023-10-14T00:04:46.665 INFO:teuthology.orchestra.run.smithi081.stdout:election_strategy: 1 2023-10-14T00:04:46.665 INFO:teuthology.orchestra.run.smithi081.stdout:0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2023-10-14T00:04:46.666 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-14T00:04:46.667 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T00:04:46.713 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e624b2c7-92dd-4594-9e39-e4d11116e328... 2023-10-14T00:04:46.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-14T00:04:46.786 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2023-10-14T00:04:46.787 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2023-10-14T00:04:46.787 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-14T00:04:46.787 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2023-10-14T00:04:46.787 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.788 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2023-10-14T00:04:46.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.788 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2023-10-14T00:04:46.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.789 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2023-10-14T00:04:46.789 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2023-10-14T00:04:46.789 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.789 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2023-10-14T00:04:46.790 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2023-10-14T00:04:46.790 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.790 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-14T00:04:46.790 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2023-10-14T00:04:46.790 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.791 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2023-10-14T00:04:46.791 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-14T00:04:46.791 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.791 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2023-10-14T00:04:46.792 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.792 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2023-10-14T00:04:46.792 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2023-10-14T00:04:46.792 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2023-10-14T00:04:46.792 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2023-10-14T00:04:46.793 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2023-10-14T00:04:46.793 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2023-10-14T00:04:46.793 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2023-10-14T00:04:46.793 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-14T00:04:46.793 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-14T00:04:46.794 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2023-10-14T00:04:46.794 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.794 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-14T00:04:46.794 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.794 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2023-10-14T00:04:46.795 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.795 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2023-10-14T00:04:46.795 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.795 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2023-10-14T00:04:46.795 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2023-10-14T00:04:46.796 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2023-10-14T00:04:46.796 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = 172.21.15.81 2023-10-14T00:04:46.796 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2023-10-14T00:04:46.796 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on pool no app = False 2023-10-14T00:04:46.796 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 5000 2023-10-14T00:04:46.797 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = e624b2c7-92dd-4594-9e39-e4d11116e328 2023-10-14T00:04:46.797 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.797 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2023-10-14T00:04:46.797 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2023-10-14T00:04:46.798 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.798 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2023-10-14T00:04:46.798 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2023-10-14T00:04:46.798 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2023-10-14T00:04:46.798 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.799 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2023-10-14T00:04:46.799 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2023-10-14T00:04:46.799 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.799 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2023-10-14T00:04:46.799 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2023-10-14T00:04:46.800 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2023-10-14T00:04:46.800 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.800 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2023-10-14T00:04:46.800 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2023-10-14T00:04:46.800 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.801 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2023-10-14T00:04:46.801 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.801 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2023-10-14T00:04:46.801 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.802 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2023-10-14T00:04:46.802 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.802 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2023-10-14T00:04:46.802 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2023-10-14T00:04:46.802 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.803 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2023-10-14T00:04:46.803 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2023-10-14T00:04:46.803 INFO:teuthology.orchestra.run.smithi081.stdout: bdev async discard = True 2023-10-14T00:04:46.803 INFO:teuthology.orchestra.run.smithi081.stdout: bdev enable discard = True 2023-10-14T00:04:46.803 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore allocator = bitmap 2023-10-14T00:04:46.804 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2023-10-14T00:04:46.804 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2023-10-14T00:04:46.804 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 1/20 2023-10-14T00:04:46.804 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 1/20 2023-10-14T00:04:46.804 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2023-10-14T00:04:46.805 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2023-10-14T00:04:46.805 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 4/10 2023-10-14T00:04:46.805 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2023-10-14T00:04:46.805 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2023-10-14T00:04:46.805 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2023-10-14T00:04:46.806 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2023-10-14T00:04:46.806 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2023-10-14T00:04:46.806 INFO:teuthology.orchestra.run.smithi081.stdout: osd shutdown pgref assert = True 2023-10-14T00:04:46.806 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.807 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2023-10-14T00:04:46.807 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2023-10-14T00:04:46.807 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2023-10-14T00:04:46.807 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2023-10-14T00:04:46.807 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2023-10-14T00:04:46.808 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2023-10-14T00:04:46.808 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2023-10-14T00:04:46.808 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2023-10-14T00:04:46.808 INFO:teuthology.orchestra.run.smithi081.stdout: debug rbd = 20 2023-10-14T00:04:46.808 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.809 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2023-10-14T00:04:46.809 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2023-10-14T00:04:46.809 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2023-10-14T00:04:46.809 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2023-10-14T00:04:46.809 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2023-10-14T00:04:46.810 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2023-10-14T00:04:46.810 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2023-10-14T00:04:46.810 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2023-10-14T00:04:46.810 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2023-10-14T00:04:46.810 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2023-10-14T00:04:46.811 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2023-10-14T00:04:46.811 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.811 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-14T00:04:46.811 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2023-10-14T00:04:46.812 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2023-10-14T00:04:46.812 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.812 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2023-10-14T00:04:46.812 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-14T00:04:46.812 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-14T00:04:46.813 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.813 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2023-10-14T00:04:46.813 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2023-10-14T00:04:46.813 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.813 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2023-10-14T00:04:46.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:04:46.814 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2023-10-14T00:04:46.814 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2023-10-14T00:04:46.814 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2023-10-14T00:04:46.814 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2023-10-14T00:04:46.815 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-14T00:04:46.815 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-14T00:04:46.815 INFO:teuthology.orchestra.run.smithi081.stdout: rbd default data pool = datapool 2023-10-14T00:04:46.816 INFO:teuthology.orchestra.run.smithi081.stdout: rbd default features = 1 2023-10-14T00:04:46.816 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2023-10-14T00:04:46.817 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-14T00:04:46.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-14T00:04:46.914 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-14T00:04:46.914 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:46.914 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-14T00:04:46.919 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:46.920 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-14T00:04:46.965 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T00:04:46.966 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:46.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-14T00:04:46.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T00:04:47.028 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:47.029 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:47.073 INFO:tasks.ceph:Setting up mon nodes... 2023-10-14T00:04:47.074 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-14T00:04:47.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T00:04:47.158 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T00:04:47.160 INFO:tasks.ceph:Setting up mds nodes... 2023-10-14T00:04:47.161 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-14T00:04:47.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-14T00:04:47.239 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-14T00:04:47.249 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-14T00:04:47.249 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-14T00:04:47.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:47.303 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:47.303 INFO:tasks.ceph:role: osd.0 2023-10-14T00:04:47.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T00:04:47.304 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-14T00:04:47.364 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T00:04:47.365 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T00:04:47.365 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T00:04:47.365 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T00:04:47.365 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T00:04:47.365 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2023-10-14T00:04:47.366 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T00:04:47.366 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T00:04:47.366 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T00:04:47.366 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T00:04:47.818 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2023-10-14T00:04:47.820 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2023-10-14T00:04:47.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:47.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:47.958 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2023-10-14T00:04:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:04:47.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:48.013 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:48.013 INFO:tasks.ceph:role: osd.1 2023-10-14T00:04:48.013 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T00:04:48.014 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-14T00:04:48.078 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T00:04:48.078 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T00:04:48.078 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T00:04:48.079 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T00:04:48.079 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T00:04:48.079 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2023-10-14T00:04:48.079 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T00:04:48.080 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T00:04:48.080 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T00:04:48.080 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T00:04:48.558 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2023-10-14T00:04:48.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2023-10-14T00:04:48.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:48.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2023-10-14T00:04:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:04:48.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:48.745 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:48.746 INFO:tasks.ceph:role: osd.2 2023-10-14T00:04:48.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T00:04:48.746 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-14T00:04:48.809 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T00:04:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T00:04:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T00:04:48.811 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T00:04:48.811 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T00:04:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2023-10-14T00:04:48.812 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T00:04:48.812 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T00:04:48.812 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T00:04:48.812 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T00:04:49.280 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2023-10-14T00:04:49.282 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2023-10-14T00:04:49.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:49.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:49.425 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2023-10-14T00:04:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:04:49.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:49.541 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:49.541+0000 7f5ac51de640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-14T00:04:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:49.545+0000 7f5ac51de640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-14T00:04:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:49.545+0000 7f5ac51de640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T00:04:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:49.545+0000 7f5ac51de640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T00:04:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:49.545+0000 7f5ac51de640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T00:04:49.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:49.545+0000 7f5ac51de640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-14T00:04:52.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:52.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:52.660 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:52.657+0000 7fa7a1237640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-14T00:04:52.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:52.657+0000 7fa7a1237640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-14T00:04:52.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:52.657+0000 7fa7a1237640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T00:04:52.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:52.657+0000 7fa7a1237640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T00:04:52.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:52.657+0000 7fa7a1237640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T00:04:52.662 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:52.661+0000 7fa7a1237640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-14T00:04:55.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:55.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:55.805 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:55.805+0000 7f485cb15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-14T00:04:55.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:55.805+0000 7f485cb15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-14T00:04:55.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:55.805+0000 7f485cb15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T00:04:55.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:55.805+0000 7f485cb15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T00:04:55.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:55.805+0000 7f485cb15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T00:04:55.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T00:04:55.805+0000 7f485cb15640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-14T00:04:58.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:58.877 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-14T00:04:58.878 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:58.878 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-14T00:04:58.932 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:58.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-14T00:04:58.984 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:58.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-14T00:04:59.036 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:59.037 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-14T00:04:59.092 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:04:59.093 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-14T00:04:59.141 INFO:tasks.ceph:Adding keys to all mons... 2023-10-14T00:04:59.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-14T00:04:59.195 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2023-10-14T00:04:59.195 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAf2ylll3mBCRAAT7QL73WsSJmlTpvn2ScQIw== 2023-10-14T00:04:59.196 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2023-10-14T00:04:59.196 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAh2yllNgNaIBAAWLdw9v9XLAgjpIa7WmfKLA== 2023-10-14T00:04:59.196 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2023-10-14T00:04:59.196 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAk2yllEZBvJxAARsjAbu77LKVWIFJ2wkGiLw== 2023-10-14T00:04:59.196 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2023-10-14T00:04:59.196 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAn2yllIEoVMBAAe3n5EoVSw1rM2j8AhT9LZg== 2023-10-14T00:04:59.197 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2023-10-14T00:04:59.197 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAf2yllh8dWDhAAhqmaw0XTdmAM37PWpDvcSw== 2023-10-14T00:04:59.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T00:04:59.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T00:04:59.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T00:04:59.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T00:04:59.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T00:04:59.575 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-14T00:04:59.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-14T00:04:59.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T00:04:59.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-14T00:04:59.734 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:59.780 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-14T00:04:59.781 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-14T00:04:59.782 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-14T00:04:59.827 INFO:tasks.ceph.mon.a:Started 2023-10-14T00:04:59.827 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-14T00:04:59.828 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-14T00:04:59.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-14T00:04:59.831 INFO:tasks.ceph.mgr.x:Started 2023-10-14T00:04:59.832 INFO:tasks.ceph:Setting crush tunables to default 2023-10-14T00:04:59.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd crush tunables default 2023-10-14T00:05:00.091 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2023-10-14T00:05:00.107 INFO:tasks.ceph:check_enable_crimson: False 2023-10-14T00:05:00.108 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-14T00:05:00.108 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:05:00.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-14T00:05:00.118 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:05:00.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-14T00:05:00.174 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:05:00.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-14T00:05:00.202 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:00.201+0000 7f39e742f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T00:05:00.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 87b96572-35f6-4cb9-a0e4-c63532c7b020 0 2023-10-14T00:05:00.445 INFO:teuthology.orchestra.run.smithi081.stdout:0 2023-10-14T00:05:00.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new d0c8a7bb-27b3-4a47-af27-af7a89c05721 1 2023-10-14T00:05:00.668 INFO:teuthology.orchestra.run.smithi081.stdout:1 2023-10-14T00:05:00.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 7da8ba88-6172-47a4-9284-457bd060469d 2 2023-10-14T00:05:00.743 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:00.741+0000 7f39e742f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T00:05:00.869 INFO:teuthology.orchestra.run.smithi081.stdout:2 2023-10-14T00:05:00.886 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-14T00:05:00.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-14T00:05:00.888 INFO:tasks.ceph.osd.0:Started 2023-10-14T00:05:00.889 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-14T00:05:00.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-14T00:05:00.891 INFO:tasks.ceph.osd.1:Started 2023-10-14T00:05:00.891 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-14T00:05:00.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-14T00:05:00.894 INFO:tasks.ceph.osd.2:Started 2023-10-14T00:05:00.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T00:05:00.939 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:00.937+0000 7f39e742f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T00:05:01.083 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:01.084 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T00:05:00.869422+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:05:01.099 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-14T00:05:01.099 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-14T00:05:01.401 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T00:05:01.506 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:05:01.505+0000 7f361401d640 -1 Falling back to public interface 2023-10-14T00:05:01.535 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.533+0000 7f39e742f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T00:05:01.537 INFO:tasks.ceph.osd.2.smithi081.stderr:2023-10-14T00:05:01.537+0000 7fa0bdbb3640 -1 Falling back to public interface 2023-10-14T00:05:01.549 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:05:01.549+0000 7ff6488c3640 -1 Falling back to public interface 2023-10-14T00:05:01.582 INFO:teuthology.misc.health.smithi081.stdout: 2023-10-14T00:05:01.583 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":5,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T00:05:00.869422+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:05:01.591 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.589+0000 7f39e742f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T00:05:01.597 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-14T00:05:01.643 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.641+0000 7f39e742f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T00:05:01.721 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.717+0000 7f39e742f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T00:05:01.770 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.769+0000 7f39e742f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T00:05:01.847 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.845+0000 7f39e742f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T00:05:01.899 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:01.897+0000 7f39e742f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T00:05:02.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:02.037+0000 7f39e742f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T00:05:02.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:02.093+0000 7f39e742f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T00:05:02.673 INFO:tasks.ceph.mgr.x.smithi081.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-14T00:05:02.674 INFO:tasks.ceph.mgr.x.smithi081.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-14T00:05:02.674 INFO:tasks.ceph.mgr.x.smithi081.stderr: from numpy import show_config as show_numpy_config 2023-10-14T00:05:02.750 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:02.749+0000 7f39e742f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T00:05:02.891 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:02.889+0000 7f39e742f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T00:05:03.044 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:03.041+0000 7f39e742f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T00:05:03.107 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:03.105+0000 7f39e742f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T00:05:03.229 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:03.229+0000 7f39e742f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T00:05:03.842 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:03.841+0000 7f39e742f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T00:05:03.916 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:03.913+0000 7f39e742f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T00:05:04.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:04.085+0000 7f39e742f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T00:05:04.158 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:04.157+0000 7f39e742f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T00:05:04.447 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:04.445+0000 7f39e742f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T00:05:04.506 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:04.505+0000 7f39e742f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T00:05:05.369 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:05:05.369+0000 7f361401d640 -1 osd.0 0 log_to_monitors true 2023-10-14T00:05:05.481 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:05:05.481+0000 7ff6488c3640 -1 osd.1 0 log_to_monitors true 2023-10-14T00:05:05.494 INFO:tasks.ceph.osd.2.smithi081.stderr:2023-10-14T00:05:05.493+0000 7fa0bdbb3640 -1 osd.2 0 log_to_monitors true 2023-10-14T00:05:05.538 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:05:05.537+0000 7f398eba2640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-14T00:05:07.900 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T00:05:08.343 INFO:teuthology.misc.health.smithi081.stdout: 2023-10-14T00:05:08.344 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":7,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T00:05:06.911926+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:05:08.370 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-14T00:05:08.756 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:05:08.753+0000 7ff645880640 -1 osd.1 0 waiting for initial osdmap 2023-10-14T00:05:08.776 INFO:tasks.ceph.osd.2.smithi081.stderr:2023-10-14T00:05:08.773+0000 7fa0b9338640 -1 osd.2 0 waiting for initial osdmap 2023-10-14T00:05:08.783 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:05:08.781+0000 7f360f7a2640 -1 osd.0 0 waiting for initial osdmap 2023-10-14T00:05:08.789 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:05:08.789+0000 7ff63c68f640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T00:05:08.797 INFO:tasks.ceph.osd.2.smithi081.stderr:2023-10-14T00:05:08.797+0000 7fa0b197f640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T00:05:08.809 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:05:08.809+0000 7f3607de9640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T00:05:14.673 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T00:05:14.961 INFO:teuthology.misc.health.smithi081.stdout: 2023-10-14T00:05:14.962 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T00:05:12.951125+0000","last_up_change":"2023-10-14T00:05:09.785799+0000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:05:10.615039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6801","nonce":290843498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6803","nonce":290843498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6807","nonce":290843498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6805","nonce":290843498}]},"public_addr":"172.21.15.81:6801/290843498","cluster_addr":"172.21.15.81:6803/290843498","heartbeat_back_addr":"172.21.15.81:6807/290843498","heartbeat_front_addr":"172.21.15.81:6805/290843498","state":["exists","up"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6817","nonce":3262473869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6819","nonce":3262473869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6823","nonce":3262473869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6821","nonce":3262473869}]},"public_addr":"172.21.15.81:6817/3262473869","cluster_addr":"172.21.15.81:6819/3262473869","heartbeat_back_addr":"172.21.15.81:6823/3262473869","heartbeat_front_addr":"172.21.15.81:6821/3262473869","state":["exists","up"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6809","nonce":1167487599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6811","nonce":1167487599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6815","nonce":1167487599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6813","nonce":1167487599}]},"public_addr":"172.21.15.81:6809/1167487599","cluster_addr":"172.21.15.81:6811/1167487599","heartbeat_back_addr":"172.21.15.81:6815/1167487599","heartbeat_front_addr":"172.21.15.81:6813/1167487599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.415523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.487442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.468257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:05:14.978 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-14T00:05:14.978 INFO:tasks.ceph:Creating RBD pool 2023-10-14T00:05:14.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-14T00:05:15.972 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'rbd' created 2023-10-14T00:05:15.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-14T00:05:16.980 INFO:teuthology.orchestra.run.smithi081.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-14T00:05:17.008 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-14T00:05:17.008 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-14T00:05:17.009 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-14T00:05:17.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-14T00:05:17.010 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-14T00:05:17.338 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:17.355 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3990506546},{"type":"v1","addr":"172.21.15.81:6825","nonce":3990506546}]},"active_addr":"172.21.15.81:6825/3990506546","active_change":"2023-10-14T00:05:04.528837+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":3796796115}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2320454148}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":198198847}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1706289227}]}]} 2023-10-14T00:05:17.358 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-14T00:05:17.358 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-14T00:05:17.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T00:05:17.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:17.647 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":13,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T00:05:16.973678+0000","last_up_change":"2023-10-14T00:05:09.785799+0000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:05:10.615039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-14T00:05:15.306730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6801","nonce":290843498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6803","nonce":290843498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6807","nonce":290843498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6805","nonce":290843498}]},"public_addr":"172.21.15.81:6801/290843498","cluster_addr":"172.21.15.81:6803/290843498","heartbeat_back_addr":"172.21.15.81:6807/290843498","heartbeat_front_addr":"172.21.15.81:6805/290843498","state":["exists","up"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6817","nonce":3262473869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6819","nonce":3262473869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6823","nonce":3262473869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6821","nonce":3262473869}]},"public_addr":"172.21.15.81:6817/3262473869","cluster_addr":"172.21.15.81:6819/3262473869","heartbeat_back_addr":"172.21.15.81:6823/3262473869","heartbeat_front_addr":"172.21.15.81:6821/3262473869","state":["exists","up"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6809","nonce":1167487599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6811","nonce":1167487599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6815","nonce":1167487599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6813","nonce":1167487599}]},"public_addr":"172.21.15.81:6809/1167487599","cluster_addr":"172.21.15.81:6811/1167487599","heartbeat_back_addr":"172.21.15.81:6815/1167487599","heartbeat_front_addr":"172.21.15.81:6813/1167487599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.415523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.487442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.468257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:05:17.663 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-14T00:05:17.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T00:05:17.934 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:17.934 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":13,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T00:05:16.973678+0000","last_up_change":"2023-10-14T00:05:09.785799+0000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:05:10.615039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-14T00:05:15.306730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6801","nonce":290843498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6803","nonce":290843498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6807","nonce":290843498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6805","nonce":290843498}]},"public_addr":"172.21.15.81:6801/290843498","cluster_addr":"172.21.15.81:6803/290843498","heartbeat_back_addr":"172.21.15.81:6807/290843498","heartbeat_front_addr":"172.21.15.81:6805/290843498","state":["exists","up"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6817","nonce":3262473869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6819","nonce":3262473869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6823","nonce":3262473869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6821","nonce":3262473869}]},"public_addr":"172.21.15.81:6817/3262473869","cluster_addr":"172.21.15.81:6819/3262473869","heartbeat_back_addr":"172.21.15.81:6823/3262473869","heartbeat_front_addr":"172.21.15.81:6821/3262473869","state":["exists","up"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6809","nonce":1167487599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6811","nonce":1167487599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6815","nonce":1167487599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6813","nonce":1167487599}]},"public_addr":"172.21.15.81:6809/1167487599","cluster_addr":"172.21.15.81:6811/1167487599","heartbeat_back_addr":"172.21.15.81:6815/1167487599","heartbeat_front_addr":"172.21.15.81:6813/1167487599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.415523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.487442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.468257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:05:17.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-14T00:05:17.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-14T00:05:17.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-14T00:05:18.100 INFO:teuthology.orchestra.run.smithi081.stdout:34359738372 2023-10-14T00:05:18.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T00:05:18.105 INFO:teuthology.orchestra.run.smithi081.stdout:34359738373 2023-10-14T00:05:18.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T00:05:18.110 INFO:teuthology.orchestra.run.smithi081.stdout:34359738373 2023-10-14T00:05:18.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T00:05:18.374 INFO:teuthology.orchestra.run.smithi081.stdout:34359738371 2023-10-14T00:05:18.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-10-14T00:05:18.423 INFO:teuthology.orchestra.run.smithi081.stdout:34359738372 2023-10-14T00:05:18.428 INFO:teuthology.orchestra.run.smithi081.stdout:34359738372 2023-10-14T00:05:18.441 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-14T00:05:18.446 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-14T00:05:19.391 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T00:05:19.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T00:05:19.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T00:05:19.686 INFO:teuthology.orchestra.run.smithi081.stdout:34359738372 2023-10-14T00:05:19.708 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-10-14T00:05:19.708 DEBUG:teuthology.parallel:result is None 2023-10-14T00:05:19.736 INFO:teuthology.orchestra.run.smithi081.stdout:34359738373 2023-10-14T00:05:19.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-14T00:05:19.753 DEBUG:teuthology.parallel:result is None 2023-10-14T00:05:19.772 INFO:teuthology.orchestra.run.smithi081.stdout:34359738373 2023-10-14T00:05:19.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-14T00:05:19.788 DEBUG:teuthology.parallel:result is None 2023-10-14T00:05:19.788 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T00:05:19.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T00:05:20.085 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:20.086 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T00:05:20.101 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-14T00:05:18.539257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82552,"kb_used_data":1488,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032968,"statfs":{"total":289910292480,"available":289825759232,"internally_reserved":0,"allocated":1523712,"data_stored":1273155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.563245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986679+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T00:05:17.986679+0000","last_peered":"2023-10-14T00:05:17.986679+0000","last_clean":"2023-10-14T00:05:17.986679+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T00:05:17.986679+0000","last_undegraded":"2023-10-14T00:05:17.986679+0000","last_fullsized":"2023-10-14T00:05:17.986679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:17:51.271717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986799+0000","last_change":"2023-10-14T00:05:17.004867+0000","last_active":"2023-10-14T00:05:17.986799+0000","last_peered":"2023-10-14T00:05:17.986799+0000","last_clean":"2023-10-14T00:05:17.986799+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T00:05:17.986799+0000","last_undegraded":"2023-10-14T00:05:17.986799+0000","last_fullsized":"2023-10-14T00:05:17.986799+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:48:27.138266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986850+0000","last_change":"2023-10-14T00:05:17.002899+0000","last_active":"2023-10-14T00:05:17.986850+0000","last_peered":"2023-10-14T00:05:17.986850+0000","last_clean":"2023-10-14T00:05:17.986850+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T00:05:17.986850+0000","last_undegraded":"2023-10-14T00:05:17.986850+0000","last_fullsized":"2023-10-14T00:05:17.986850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:43:16.014296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986757+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T00:05:17.986757+0000","last_peered":"2023-10-14T00:05:17.986757+0000","last_clean":"2023-10-14T00:05:17.986757+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T00:05:17.986757+0000","last_undegraded":"2023-10-14T00:05:17.986757+0000","last_fullsized":"2023-10-14T00:05:17.986757+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:37:32.329836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.002455+0000","last_change":"2023-10-14T00:05:17.000382+0000","last_active":"2023-10-14T00:05:17.002455+0000","last_peered":"2023-10-14T00:05:17.002455+0000","last_clean":"2023-10-14T00:05:17.002455+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T00:05:17.002455+0000","last_undegraded":"2023-10-14T00:05:17.002455+0000","last_fullsized":"2023-10-14T00:05:17.002455+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:47:56.597303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.004889+0000","last_change":"2023-10-14T00:05:16.999347+0000","last_active":"2023-10-14T00:05:17.004889+0000","last_peered":"2023-10-14T00:05:17.004889+0000","last_clean":"2023-10-14T00:05:17.004889+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T00:05:17.004889+0000","last_undegraded":"2023-10-14T00:05:17.004889+0000","last_fullsized":"2023-10-14T00:05:17.004889+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:29:57.552899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.005347+0000","last_change":"2023-10-14T00:05:17.001059+0000","last_active":"2023-10-14T00:05:17.005347+0000","last_peered":"2023-10-14T00:05:17.005347+0000","last_clean":"2023-10-14T00:05:17.005347+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T00:05:17.005347+0000","last_undegraded":"2023-10-14T00:05:17.005347+0000","last_fullsized":"2023-10-14T00:05:17.005347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:21:14.370653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986678+0000","last_change":"2023-10-14T00:05:17.005268+0000","last_active":"2023-10-14T00:05:17.986678+0000","last_peered":"2023-10-14T00:05:17.986678+0000","last_clean":"2023-10-14T00:05:17.986678+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T00:05:17.986678+0000","last_undegraded":"2023-10-14T00:05:17.986678+0000","last_fullsized":"2023-10-14T00:05:17.986678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:44:40.852686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:16.998139+0000","last_change":"2023-10-14T00:05:12.384839+0000","last_active":"2023-10-14T00:05:16.998139+0000","last_peered":"2023-10-14T00:05:16.998139+0000","last_clean":"2023-10-14T00:05:16.998139+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T00:05:16.998139+0000","last_undegraded":"2023-10-14T00:05:16.998139+0000","last_fullsized":"2023-10-14T00:05:16.998139+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_clean_scrub_stamp":"2023-10-14T00:05:10.931569+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:10:36.076607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":110592,"data_stored":30550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":620918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T00:05:20.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T00:05:20.356 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T00:05:20.372 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-14T00:05:18.539257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82552,"kb_used_data":1488,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032968,"statfs":{"total":289910292480,"available":289825759232,"internally_reserved":0,"allocated":1523712,"data_stored":1273155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.563245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986679+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T00:05:17.986679+0000","last_peered":"2023-10-14T00:05:17.986679+0000","last_clean":"2023-10-14T00:05:17.986679+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T00:05:17.986679+0000","last_undegraded":"2023-10-14T00:05:17.986679+0000","last_fullsized":"2023-10-14T00:05:17.986679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:17:51.271717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986799+0000","last_change":"2023-10-14T00:05:17.004867+0000","last_active":"2023-10-14T00:05:17.986799+0000","last_peered":"2023-10-14T00:05:17.986799+0000","last_clean":"2023-10-14T00:05:17.986799+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T00:05:17.986799+0000","last_undegraded":"2023-10-14T00:05:17.986799+0000","last_fullsized":"2023-10-14T00:05:17.986799+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:48:27.138266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986850+0000","last_change":"2023-10-14T00:05:17.002899+0000","last_active":"2023-10-14T00:05:17.986850+0000","last_peered":"2023-10-14T00:05:17.986850+0000","last_clean":"2023-10-14T00:05:17.986850+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T00:05:17.986850+0000","last_undegraded":"2023-10-14T00:05:17.986850+0000","last_fullsized":"2023-10-14T00:05:17.986850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:43:16.014296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986757+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T00:05:17.986757+0000","last_peered":"2023-10-14T00:05:17.986757+0000","last_clean":"2023-10-14T00:05:17.986757+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T00:05:17.986757+0000","last_undegraded":"2023-10-14T00:05:17.986757+0000","last_fullsized":"2023-10-14T00:05:17.986757+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:37:32.329836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.002455+0000","last_change":"2023-10-14T00:05:17.000382+0000","last_active":"2023-10-14T00:05:17.002455+0000","last_peered":"2023-10-14T00:05:17.002455+0000","last_clean":"2023-10-14T00:05:17.002455+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T00:05:17.002455+0000","last_undegraded":"2023-10-14T00:05:17.002455+0000","last_fullsized":"2023-10-14T00:05:17.002455+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:47:56.597303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.004889+0000","last_change":"2023-10-14T00:05:16.999347+0000","last_active":"2023-10-14T00:05:17.004889+0000","last_peered":"2023-10-14T00:05:17.004889+0000","last_clean":"2023-10-14T00:05:17.004889+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T00:05:17.004889+0000","last_undegraded":"2023-10-14T00:05:17.004889+0000","last_fullsized":"2023-10-14T00:05:17.004889+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:29:57.552899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.005347+0000","last_change":"2023-10-14T00:05:17.001059+0000","last_active":"2023-10-14T00:05:17.005347+0000","last_peered":"2023-10-14T00:05:17.005347+0000","last_clean":"2023-10-14T00:05:17.005347+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T00:05:17.005347+0000","last_undegraded":"2023-10-14T00:05:17.005347+0000","last_fullsized":"2023-10-14T00:05:17.005347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:21:14.370653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:05:17.986678+0000","last_change":"2023-10-14T00:05:17.005268+0000","last_active":"2023-10-14T00:05:17.986678+0000","last_peered":"2023-10-14T00:05:17.986678+0000","last_clean":"2023-10-14T00:05:17.986678+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T00:05:17.986678+0000","last_undegraded":"2023-10-14T00:05:17.986678+0000","last_fullsized":"2023-10-14T00:05:17.986678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:44:40.852686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:05:16.998139+0000","last_change":"2023-10-14T00:05:12.384839+0000","last_active":"2023-10-14T00:05:16.998139+0000","last_peered":"2023-10-14T00:05:16.998139+0000","last_clean":"2023-10-14T00:05:16.998139+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T00:05:16.998139+0000","last_undegraded":"2023-10-14T00:05:16.998139+0000","last_fullsized":"2023-10-14T00:05:16.998139+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_clean_scrub_stamp":"2023-10-14T00:05:10.931569+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:10:36.076607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":110592,"data_stored":30550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":620918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T00:05:20.373 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T00:05:20.373 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-14T00:05:20.374 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-14T00:05:20.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-14T00:05:20.679 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T00:05:20.679 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-14T00:05:20.696 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-14T00:05:20.696 INFO:teuthology.run_tasks:Running task exec... 2023-10-14T00:05:20.707 INFO:teuthology.task.exec:Executing custom commands... 2023-10-14T00:05:20.708 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T00:05:20.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-14T00:05:21.071 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'datapool' created 2023-10-14T00:05:21.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-14T00:05:24.119 INFO:teuthology.run_tasks:Running task install... 2023-10-14T00:05:24.136 DEBUG:teuthology.task.install:project ceph 2023-10-14T00:05:24.136 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}} 2023-10-14T00:05:24.136 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'} 2023-10-14T00:05:24.136 INFO:teuthology.task.install:Using flavor: default 2023-10-14T00:05:24.142 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-14T00:05:24.142 INFO:teuthology.task.install:extra packages: [] 2023-10-14T00:05:24.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-key list | grep Ceph 2023-10-14T00:05:24.211 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-14T00:05:24.241 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-14T00:05:24.241 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph.com (release key) 2023-10-14T00:05:24.242 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-14T00:05:24.242 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-14T00:05:24.245 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:05:24.469 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default/ 2023-10-14T00:05:24.470 INFO:teuthology.task.install.deb:Package version is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:05:24.575 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:05:24.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-14T00:05:24.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2023-10-14T00:05:24.775 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-14T00:05:24.775 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-14T00:05:24.800 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-14T00:05:24.814 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy InRelease 2023-10-14T00:05:24.835 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:5 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release 2023-10-14T00:05:24.851 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-14T00:05:24.856 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-14T00:05:26.028 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-14T00:05:26.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6724-g01fff6a7-1jammy cephadm=18.0.0-6724-g01fff6a7-1jammy ceph-mds=18.0.0-6724-g01fff6a7-1jammy ceph-mgr=18.0.0-6724-g01fff6a7-1jammy ceph-common=18.0.0-6724-g01fff6a7-1jammy ceph-fuse=18.0.0-6724-g01fff6a7-1jammy ceph-test=18.0.0-6724-g01fff6a7-1jammy ceph-volume=18.0.0-6724-g01fff6a7-1jammy radosgw=18.0.0-6724-g01fff6a7-1jammy python3-rados=18.0.0-6724-g01fff6a7-1jammy python3-rgw=18.0.0-6724-g01fff6a7-1jammy python3-cephfs=18.0.0-6724-g01fff6a7-1jammy python3-rbd=18.0.0-6724-g01fff6a7-1jammy libcephfs2=18.0.0-6724-g01fff6a7-1jammy libcephfs-dev=18.0.0-6724-g01fff6a7-1jammy librados2=18.0.0-6724-g01fff6a7-1jammy librbd1=18.0.0-6724-g01fff6a7-1jammy rbd-fuse=18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:05:26.100 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-14T00:05:26.276 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2023-10-14T00:05:26.277 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2023-10-14T00:05:26.435 INFO:teuthology.orchestra.run.smithi081.stdout:ceph is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.436 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-common is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.436 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-fuse is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.437 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mds is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.437 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mgr is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.438 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-test is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.438 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-volume is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.439 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.439 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-dev is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.440 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs2 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.440 INFO:teuthology.orchestra.run.smithi081.stdout:librados2 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.441 INFO:teuthology.orchestra.run.smithi081.stdout:librbd1 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.441 INFO:teuthology.orchestra.run.smithi081.stdout:python3-cephfs is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.441 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rados is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.442 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rbd is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.442 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rgw is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.442 INFO:teuthology.orchestra.run.smithi081.stdout:radosgw is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.442 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:05:26.443 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2023-10-14T00:05:26.443 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-14T00:05:26.443 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-14T00:05:26.443 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-14T00:05:26.443 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-14T00:05:26.444 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-14T00:05:26.444 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-14T00:05:26.470 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 0 newly installed, 0 to remove and 256 not upgraded. 2023-10-14T00:05:26.470 INFO:teuthology.orchestra.run.smithi081.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-14T00:05:26.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-14T00:05:26.523 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2023-10-14T00:05:26.689 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2023-10-14T00:05:26.689 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2023-10-14T00:05:26.834 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2023-10-14T00:05:26.834 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-14T00:05:26.834 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-14T00:05:26.834 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-14T00:05:26.835 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-14T00:05:26.835 INFO:teuthology.orchestra.run.smithi081.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-14T00:05:26.835 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-14T00:05:26.836 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2023-10-14T00:05:26.836 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemblk1 2023-10-14T00:05:26.837 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2023-10-14T00:05:26.837 INFO:teuthology.orchestra.run.smithi081.stdout: gnuplot gfio python-scipy 2023-10-14T00:05:26.862 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2023-10-14T00:05:26.863 INFO:teuthology.orchestra.run.smithi081.stdout: fio libpmemblk1 2023-10-14T00:05:27.042 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 2 newly installed, 0 to remove and 256 not upgraded. 2023-10-14T00:05:27.042 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 4,112 kB of archives. 2023-10-14T00:05:27.042 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-14T00:05:27.042 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-14T00:05:27.326 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-14T00:05:28.189 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 4,112 kB in 1s (4,261 kB/s) 2023-10-14T00:05:28.351 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-14T00:05:28.454 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172814 files and directories currently installed.) 2023-10-14T00:05:28.459 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-14T00:05:28.541 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-14T00:05:28.870 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package fio. 2023-10-14T00:05:28.897 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-14T00:05:28.936 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking fio (3.28-1) ... 2023-10-14T00:05:29.530 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-14T00:05:29.659 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up fio (3.28-1) ... 2023-10-14T00:05:30.153 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-14T00:05:31.229 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-14T00:05:31.672 INFO:teuthology.orchestra.run.smithi081.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-14T00:05:31.674 DEBUG:teuthology.parallel:result is None 2023-10-14T00:05:31.675 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:05:31.827 DEBUG:teuthology.orchestra.run.smithi081:> dpkg-query -W -f '${Version}' ceph 2023-10-14T00:05:31.846 INFO:teuthology.orchestra.run.smithi081.stdout:18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:05:31.847 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:05:31.847 INFO:teuthology.task.install:The correct ceph version 18.0.0-6724-g01fff6a7-1jammy is installed. 2023-10-14T00:05:31.847 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-14T00:05:31.848 INFO:teuthology.run_tasks:Running task workunit... 2023-10-14T00:05:31.861 INFO:tasks.workunit:Pulling workunits from ref 735f68d7bf91d643813a8420fc2b11441f4bd8c4 2023-10-14T00:05:31.861 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-14T00:05:31.862 DEBUG:teuthology.orchestra.run.smithi081:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-14T00:05:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:05:31.895 INFO:teuthology.orchestra.run.smithi081.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-14T00:05:31.896 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-14T00:05:31.950 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-14T00:05:31.951 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-14T00:05:31.998 INFO:tasks.workunit:timeout=3h 2023-10-14T00:05:31.998 INFO:tasks.workunit:cleanup=True 2023-10-14T00:05:31.999 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/idryomov/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 735f68d7bf91d643813a8420fc2b11441f4bd8c4 2023-10-14T00:05:32.047 INFO:tasks.workunit.client.0.smithi081.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-14T00:07:01.911 INFO:tasks.workunit.client.0.smithi081.stderr:Note: switching to '735f68d7bf91d643813a8420fc2b11441f4bd8c4'. 2023-10-14T00:07:01.911 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.912 INFO:tasks.workunit.client.0.smithi081.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-14T00:07:01.912 INFO:tasks.workunit.client.0.smithi081.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-14T00:07:01.912 INFO:tasks.workunit.client.0.smithi081.stderr:state without impacting any branches by switching back to a branch. 2023-10-14T00:07:01.912 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.912 INFO:tasks.workunit.client.0.smithi081.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-14T00:07:01.913 INFO:tasks.workunit.client.0.smithi081.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-14T00:07:01.913 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.913 INFO:tasks.workunit.client.0.smithi081.stderr: git switch -c 2023-10-14T00:07:01.913 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.913 INFO:tasks.workunit.client.0.smithi081.stderr:Or undo this operation with: 2023-10-14T00:07:01.914 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.914 INFO:tasks.workunit.client.0.smithi081.stderr: git switch - 2023-10-14T00:07:01.914 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.914 INFO:tasks.workunit.client.0.smithi081.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-14T00:07:01.914 INFO:tasks.workunit.client.0.smithi081.stderr: 2023-10-14T00:07:01.915 INFO:tasks.workunit.client.0.smithi081.stderr:HEAD is now at 735f68d7bf9 Merge branch 'wip-62891' of https://github.com/ajarr/ceph into HEAD 2023-10-14T00:07:01.917 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-14T00:07:01.962 INFO:tasks.workunit.client.0.smithi081.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-14T00:07:01.965 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-14T00:07:01.966 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-14T00:07:02.708 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-14T00:07:02.784 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-14T00:07:02.838 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-14T00:07:02.839 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-14T00:07:02.839 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-14T00:07:02.883 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-14T00:07:02.887 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-10-14T00:07:02.887 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-14T00:07:02.934 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-14T00:07:02.935 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-14T00:07:02.936 DEBUG:teuthology.orchestra.run.smithi081:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-14T00:07:02.986 INFO:tasks.workunit.client.0.smithi081.stderr:+ POOL=rbd 2023-10-14T00:07:02.986 INFO:tasks.workunit.client.0.smithi081.stderr:+ IMAGE_PREFIX=image 2023-10-14T00:07:02.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ NUM_IMAGES=20 2023-10-14T00:07:02.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ RUN_TIME=3600 2023-10-14T00:07:02.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror pool enable rbd image 2023-10-14T00:07:03.032 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-14T00:07:03.067 INFO:tasks.workunit.client.0.smithi081.stdout:1165ab61-3458-4cf5-b295-6f23b6b88493 2023-10-14T00:07:03.070 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i = 1 )) 2023-10-14T00:07:03.070 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:03.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-14T00:07:03.113 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-14T00:07:04.006 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:04.017 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-14T00:07:04.080 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:04.081 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:04.081 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-14T00:07:04.128 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-14T00:07:05.005 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:05.017 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-14T00:07:05.075 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:05.076 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:05.076 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-14T00:07:05.122 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-14T00:07:06.022 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:06.036 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-14T00:07:06.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:06.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:06.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-14T00:07:06.146 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-14T00:07:07.027 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:07.041 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-14T00:07:07.100 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:07.101 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:07.101 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-14T00:07:07.150 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-14T00:07:08.039 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:08.051 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-14T00:07:08.114 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:08.114 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:08.114 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-14T00:07:08.163 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-14T00:07:09.049 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:09.065 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-14T00:07:09.124 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:09.125 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:09.125 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-14T00:07:09.171 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-14T00:07:10.054 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:10.067 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-14T00:07:10.128 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:10.129 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:10.129 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-14T00:07:10.178 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-14T00:07:11.065 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:11.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-14T00:07:11.137 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:11.137 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:11.137 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-14T00:07:11.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-14T00:07:12.068 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:12.080 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-14T00:07:12.143 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:12.144 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:12.144 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-14T00:07:12.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-14T00:07:13.076 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:13.088 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-14T00:07:13.148 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:13.148 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:13.149 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-14T00:07:13.199 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-14T00:07:14.086 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:14.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-14T00:07:14.160 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:14.160 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:14.160 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-14T00:07:14.208 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-14T00:07:15.094 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:15.108 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-14T00:07:15.169 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:15.170 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:15.170 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-14T00:07:15.220 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-14T00:07:16.092 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:16.107 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-14T00:07:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-14T00:07:16.216 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-14T00:07:17.104 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:17.118 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-14T00:07:17.179 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:17.180 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:17.180 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-14T00:07:17.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-14T00:07:18.112 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:18.124 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-14T00:07:18.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:18.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:18.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-14T00:07:18.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-14T00:07:19.124 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:19.139 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-14T00:07:19.200 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:19.201 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:19.201 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-14T00:07:19.249 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-14T00:07:20.131 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:20.143 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-14T00:07:20.202 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:20.202 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:20.202 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-14T00:07:20.251 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-14T00:07:21.151 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:21.163 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-14T00:07:21.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:21.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:21.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-14T00:07:21.275 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-14T00:07:22.155 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:22.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-14T00:07:22.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:22.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:22.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-14T00:07:22.280 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-14T00:07:23.160 INFO:tasks.workunit.client.0.smithi081.stdout:Mirroring enabled 2023-10-14T00:07:23.172 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-14T00:07:23.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.233 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.233 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i = 1 )) 2023-10-14T00:07:23.233 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.233 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image1 2023-10-14T00:07:23.366 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-14T00:07:23.366 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.367 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.367 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.367 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image2 2023-10-14T00:07:23.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-14T00:07:23.438 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.438 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.438 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.438 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image3 2023-10-14T00:07:23.509 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-14T00:07:23.509 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.509 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.509 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.510 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image4 2023-10-14T00:07:23.593 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-14T00:07:23.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.595 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.595 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image5 2023-10-14T00:07:23.677 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-14T00:07:23.677 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.677 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.677 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.678 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image6 2023-10-14T00:07:23.746 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-14T00:07:23.746 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.746 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.746 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.747 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image7 2023-10-14T00:07:23.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-14T00:07:23.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.852 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image8 2023-10-14T00:07:23.968 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-14T00:07:23.968 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:23.968 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:23.968 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:23.969 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image9 2023-10-14T00:07:24.094 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-14T00:07:24.094 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:24.094 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:24.095 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:24.095 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image10 2023-10-14T00:07:24.243 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-14T00:07:24.243 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:24.244 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:24.244 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:24.251 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image11 2023-10-14T00:07:24.359 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-14T00:07:24.360 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:24.360 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:24.360 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:24.360 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image12 2023-10-14T00:07:24.672 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-14T00:07:24.673 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:24.673 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:24.673 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:24.673 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image13 2023-10-14T00:07:24.789 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-14T00:07:24.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:24.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:24.790 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image14 2023-10-14T00:07:24.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:24.934 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-14T00:07:24.934 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:24.934 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:24.934 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:24.935 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image15 2023-10-14T00:07:25.053 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-14T00:07:25.054 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:25.054 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:25.055 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image16 2023-10-14T00:07:25.055 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:25.205 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-14T00:07:25.205 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:25.206 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:25.206 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:25.209 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image17 2023-10-14T00:07:25.327 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-14T00:07:25.328 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:25.328 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:25.328 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:25.328 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image18 2023-10-14T00:07:25.479 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-14T00:07:25.480 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:25.480 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:25.480 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:25.480 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image19 2023-10-14T00:07:25.664 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-14T00:07:25.665 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:25.665 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:25.665 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:25.665 INFO:tasks.workunit.client.0.smithi081.stderr:++ sudo rbd device map rbd/image20 2023-10-14T00:07:25.798 INFO:tasks.workunit.client.0.smithi081.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-14T00:07:25.798 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T00:07:25.798 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 20 )) 2023-10-14T00:07:25.798 INFO:tasks.workunit.client.0.smithi081.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:07:25.799 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:07:25.799 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242045 2023-10-14T00:07:25.799 INFO:tasks.workunit.client.0.smithi081.stderr:+ END_TIME=1697245645 2023-10-14T00:07:25.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR= 2023-10-14T00:07:25.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR= 2023-10-14T00:07:25.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:25.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n '' ]] 2023-10-14T00:07:25.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:07:33.642 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:07:33.637+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:07:35.803 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:35.804 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:35.804 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:35.804 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:07:36.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/198198847 2023-10-14T00:07:36.783 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:07:36.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242056 2023-10-14T00:07:36.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:36.791 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/198198847 ]] 2023-10-14T00:07:36.791 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/198198847 != '' ]] 2023-10-14T00:07:36.791 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/198198847 2023-10-14T00:07:38.984 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/198198847 until 2023-10-14T01:07:38.251236+0000 (3600 sec) 2023-10-14T00:07:39.012 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:07:39.014 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/198198847 2023-10-14T00:07:39.707 INFO:tasks.workunit.client.0.smithi081.stderr:listed 1 entries 2023-10-14T00:07:39.733 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/198198847 2023-10-14T00:07:39.733 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:07:49.734 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:07:49.738 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:49.738 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:49.742 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:50.631 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/198198847 2023-10-14T00:07:50.632 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:07:50.632 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242070 2023-10-14T00:07:50.632 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:50.633 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/198198847 ]] 2023-10-14T00:07:50.633 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/198198847 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\8\1\9\8\8\4\7 ]] 2023-10-14T00:07:50.633 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:08:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:08:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:08:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:08:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde1aa800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:08:00.635 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:08:00.635 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:00.645 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:00.650 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:01.528 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/198198847 2023-10-14T00:08:01.528 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:08:01.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242081 2023-10-14T00:08:01.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:01.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/198198847 ]] 2023-10-14T00:08:01.530 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/198198847 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\8\1\9\8\8\4\7 ]] 2023-10-14T00:08:01.530 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:08:11.532 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:11.533 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:08:11.536 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:11.536 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:12.376 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:08:12.376 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:08:12.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242092 2023-10-14T00:08:12.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:12.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:08:12.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\8\1\9\8\8\4\7 ]] 2023-10-14T00:08:12.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3807701574 2023-10-14T00:08:14.027 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3807701574 until 2023-10-14T01:08:13.228439+0000 (3600 sec) 2023-10-14T00:08:14.063 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3807701574 2023-10-14T00:08:14.064 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:08:14.833 INFO:tasks.workunit.client.0.smithi081.stderr:listed 2 entries 2023-10-14T00:08:14.868 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:08:14.868 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:08:24.871 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:24.875 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:24.882 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:24.891 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:08:25.996 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:08:25.996 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:08:25.997 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242105 2023-10-14T00:08:25.997 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:25.997 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:08:25.997 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:08:25.997 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:08:36.001 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:36.002 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:08:36.002 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:36.008 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:36.862 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:08:36.862 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:08:36.863 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242116 2023-10-14T00:08:36.863 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:36.863 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:08:36.864 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:08:36.864 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:08:46.869 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:08:46.869 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:46.869 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:46.873 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:47.697 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:08:47.697 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:08:47.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242127 2023-10-14T00:08:47.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:47.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:08:47.699 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:08:47.699 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:08:57.702 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:08:57.703 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:57.704 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:57.704 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:58.522 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:08:58.522 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:08:58.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242138 2023-10-14T00:08:58.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:58.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:08:58.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:08:58.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:09:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.004+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.004+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.004+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde8dd900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.004+0000 7f3980385640 -1 librbd::ImageState: 0x563dde8ddb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.008+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.008+0000 7f3980385640 -1 librbd::ImageState: 0x563dde063200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.008+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.008+0000 7f3980385640 -1 librbd::ImageState: 0x563ddeb6be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.012+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.012+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.016+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.016+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde8dd900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.020+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.020+0000 7f3980385640 -1 librbd::ImageState: 0x563ddec68580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.024+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.024+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.031 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.028+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.031 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.028+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde8ddc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.032+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.032+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde8ddb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.036+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.036+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde063200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.040+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.040+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.044+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.044+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde049d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.048+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.048+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.052+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.052+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.062 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.056+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.062 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.056+0000 7f3980385640 -1 librbd::ImageState: 0x563dde8ddc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.060+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.060+0000 7f3980385640 -1 librbd::ImageState: 0x563dde8dda00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.064+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.064+0000 7f3980385640 -1 librbd::ImageState: 0x563dde8dde80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.068+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.068+0000 7f3980385640 -1 librbd::ImageState: 0x563ddec68480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.076+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:00.076+0000 7f3980385640 -1 librbd::ImageState: 0x563ddec68580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:08.531 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:09:08.532 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:08.532 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:08.556 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:09.405 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:09:09.405 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:09:09.406 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242149 2023-10-14T00:09:09.406 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:09.406 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:09:09.407 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:09:09.407 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:09:19.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:19.423 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:19.426 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:09:19.434 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:20.242 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3807701574 2023-10-14T00:09:20.243 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:09:20.244 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242160 2023-10-14T00:09:20.244 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:20.244 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3807701574 ]] 2023-10-14T00:09:20.244 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3807701574 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:09:20.245 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:09:30.245 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:09:30.246 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:30.255 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:30.273 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:31.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:09:31.274 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:09:31.275 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242171 2023-10-14T00:09:31.275 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:31.275 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3424877767 ]] 2023-10-14T00:09:31.276 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3424877767 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\8\0\7\7\0\1\5\7\4 ]] 2023-10-14T00:09:31.276 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3424877767 2023-10-14T00:09:33.058 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3424877767 until 2023-10-14T01:09:32.319612+0000 (3600 sec) 2023-10-14T00:09:33.092 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3424877767 2023-10-14T00:09:33.092 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:09:33.822 INFO:tasks.workunit.client.0.smithi081.stderr:listed 3 entries 2023-10-14T00:09:33.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:09:33.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:09:43.854 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:09:43.855 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:43.855 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:43.857 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:44.786 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:09:44.787 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:09:44.788 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242184 2023-10-14T00:09:44.788 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:44.788 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3424877767 ]] 2023-10-14T00:09:44.788 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3424877767 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\2\4\8\7\7\7\6\7 ]] 2023-10-14T00:09:44.789 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:09:51.418 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:09:51.411+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:09:54.789 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:54.790 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:09:54.791 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:54.794 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:55.649 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:09:55.650 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:09:55.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242195 2023-10-14T00:09:55.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:55.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3424877767 ]] 2023-10-14T00:09:55.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3424877767 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\2\4\8\7\7\7\6\7 ]] 2023-10-14T00:09:55.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:10:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:59.998+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:09:59.998+0000 7f3981387640 -1 librbd::ImageState: 0x563dde240900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.002+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.002+0000 7f3981387640 -1 librbd::ImageState: 0x563dde240400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.006+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.006+0000 7f3981387640 -1 librbd::ImageState: 0x563dde171600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.019 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.014+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.019 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.014+0000 7f3981387640 -1 librbd::ImageState: 0x563dde8dc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.022+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.022+0000 7f3981387640 -1 librbd::ImageState: 0x563ddea1e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.026+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.026+0000 7f3981387640 -1 librbd::ImageState: 0x563ddec68480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.030+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.030+0000 7f3981387640 -1 librbd::ImageState: 0x563dde062980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.038+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.038+0000 7f3981387640 -1 librbd::ImageState: 0x563dde171d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.042+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.042+0000 7f3981387640 -1 librbd::ImageState: 0x563ddec2f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.050+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.050+0000 7f3981387640 -1 librbd::ImageState: 0x563ddec2fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.062+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.062+0000 7f3981387640 -1 librbd::ImageState: 0x563dde8dc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.066+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.066+0000 7f3981387640 -1 librbd::ImageState: 0x563ddeb6a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.074+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.074+0000 7f3981387640 -1 librbd::ImageState: 0x563dde240380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.078+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.078+0000 7f3981387640 -1 librbd::ImageState: 0x563dde240500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.086+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.086+0000 7f3981387640 -1 librbd::ImageState: 0x563ddec2f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.090+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.090+0000 7f3981387640 -1 librbd::ImageState: 0x563ddec2f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.103 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.098+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.103 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.098+0000 7f3981387640 -1 librbd::ImageState: 0x563dde171d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.110 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.102+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.110 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.102+0000 7f3981387640 -1 librbd::ImageState: 0x563ddea1fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.110+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.110+0000 7f3981387640 -1 librbd::ImageState: 0x563dde240680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.121 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.114+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.121 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:10:00.114+0000 7f3981387640 -1 librbd::ImageState: 0x563dde240300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:01.308 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:10:01.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:10:01.308 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:10:01.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:10:01.309 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:10:01.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:10:01.309 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:10:01.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:10:05.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:10:05.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:05.659 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:05.671 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:06.568 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:10:06.569 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:10:06.569 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242206 2023-10-14T00:10:06.569 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:06.570 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3424877767 ]] 2023-10-14T00:10:06.570 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3424877767 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\2\4\8\7\7\7\6\7 ]] 2023-10-14T00:10:06.570 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:10:10.058 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:10:10.054+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:10:16.575 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:16.576 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:10:16.582 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:16.592 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:17.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:10:17.504 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:10:17.504 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242217 2023-10-14T00:10:17.505 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:17.505 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3424877767 ]] 2023-10-14T00:10:17.505 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3424877767 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\2\4\8\7\7\7\6\7 ]] 2023-10-14T00:10:17.506 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:10:27.506 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:27.507 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:10:27.507 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:27.538 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:28.363 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3424877767 2023-10-14T00:10:28.364 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:10:28.364 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242228 2023-10-14T00:10:28.364 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:28.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3424877767 ]] 2023-10-14T00:10:28.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3424877767 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\2\4\8\7\7\7\6\7 ]] 2023-10-14T00:10:28.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:10:34.999 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:10:34.998+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:10:38.372 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:38.373 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:38.376 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:38.377 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:10:39.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:10:39.233 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:10:39.234 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242239 2023-10-14T00:10:39.234 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:39.234 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1294444516 ]] 2023-10-14T00:10:39.234 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1294444516 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\2\4\8\7\7\7\6\7 ]] 2023-10-14T00:10:39.235 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1294444516 2023-10-14T00:10:41.017 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1294444516 until 2023-10-14T01:10:40.246746+0000 (3600 sec) 2023-10-14T00:10:41.062 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:10:41.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1294444516 2023-10-14T00:10:41.807 INFO:tasks.workunit.client.0.smithi081.stderr:listed 4 entries 2023-10-14T00:10:41.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:10:41.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:10:51.843 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:51.843 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:51.846 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:51.846 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:10:52.711 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:10:52.716 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:10:52.718 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242252 2023-10-14T00:10:52.719 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:52.719 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1294444516 ]] 2023-10-14T00:10:52.719 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1294444516 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\2\9\4\4\4\4\5\1\6 ]] 2023-10-14T00:10:52.719 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:11:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.001+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.001+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddea1f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec69980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde240200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.013+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.013+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde062c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.023 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.021+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.023 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.021+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde240c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.031 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.029+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.031 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.029+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde241200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.037+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.037+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde241e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.041+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.041+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde241a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.041+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.041+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.049+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.049+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.057+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.057+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde1abb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.063 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.061+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.061+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec69580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.065+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.065+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.075 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.073+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.076 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.073+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.080 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.077+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.077+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.081+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.081+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef00080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.085+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.085+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef00280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.093+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.093+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef00480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.097+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.097+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef00e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.101 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.097+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.101 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:11:00.097+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef00d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:02.720 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:11:02.720 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:02.723 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:02.726 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:03.592 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:11:03.593 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:11:03.593 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242263 2023-10-14T00:11:03.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:03.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1294444516 ]] 2023-10-14T00:11:03.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1294444516 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\2\9\4\4\4\4\5\1\6 ]] 2023-10-14T00:11:03.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:11:13.596 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:13.596 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:11:13.609 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:13.612 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:14.585 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:11:14.600 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:11:14.601 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242274 2023-10-14T00:11:14.602 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:14.602 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1294444516 ]] 2023-10-14T00:11:14.602 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1294444516 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\2\9\4\4\4\4\5\1\6 ]] 2023-10-14T00:11:14.602 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:11:24.613 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:11:24.631 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:24.643 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:24.643 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:25.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:11:25.500 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:11:25.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242285 2023-10-14T00:11:25.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:25.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1294444516 ]] 2023-10-14T00:11:25.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1294444516 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\2\9\4\4\4\4\5\1\6 ]] 2023-10-14T00:11:25.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:11:35.503 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:11:35.503 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:35.504 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:35.505 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:36.453 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1294444516 2023-10-14T00:11:36.454 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:11:36.454 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242296 2023-10-14T00:11:36.454 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:36.455 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1294444516 ]] 2023-10-14T00:11:36.455 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1294444516 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\2\9\4\4\4\4\5\1\6 ]] 2023-10-14T00:11:36.455 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:11:46.457 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:11:46.472 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:46.473 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:46.474 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:47.351 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:11:47.352 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:11:47.352 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242307 2023-10-14T00:11:47.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:47.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/991631890 ]] 2023-10-14T00:11:47.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/991631890 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\2\9\4\4\4\4\5\1\6 ]] 2023-10-14T00:11:47.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/991631890 2023-10-14T00:11:49.078 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/991631890 until 2023-10-14T01:11:48.286453+0000 (3600 sec) 2023-10-14T00:11:49.129 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:11:49.132 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/991631890 2023-10-14T00:11:49.954 INFO:tasks.workunit.client.0.smithi081.stderr:listed 5 entries 2023-10-14T00:11:49.984 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:11:49.984 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:11:59.985 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:11:59.989 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:59.991 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:00.004 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8e580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.013+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.013+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef01880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.017+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.019 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.017+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddec68180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.021+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.025+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef01680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.037+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.037+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef01380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.041+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.041+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.053+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.053+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.057+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.057+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.061+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.065 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.061+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.069+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.069+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.087 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.085+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.085+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef00c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.105+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.105+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.113+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.113+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.129 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.125+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.129 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.125+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.135 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.133+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.135 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.133+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef17180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.150 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.149+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.150 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.149+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef17380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.156 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.153+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.156 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.153+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef17580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.161 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.157+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.161 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.157+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef17780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.165 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.161+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.166 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:12:00.161+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef17980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.913 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:12:00.917 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:12:00.918 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242320 2023-10-14T00:12:00.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:00.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/991631890 ]] 2023-10-14T00:12:00.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/991631890 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\9\9\1\6\3\1\8\9\0 ]] 2023-10-14T00:12:00.920 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:12:10.920 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:10.921 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:10.921 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:12:10.933 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:11.805 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:12:11.806 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:12:11.806 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242331 2023-10-14T00:12:11.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:11.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/991631890 ]] 2023-10-14T00:12:11.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/991631890 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\9\9\1\6\3\1\8\9\0 ]] 2023-10-14T00:12:11.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:12:21.808 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:12:21.808 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:21.808 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:21.817 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:22.633 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:12:22.644 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:12:22.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242342 2023-10-14T00:12:22.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:22.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/991631890 ]] 2023-10-14T00:12:22.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/991631890 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\9\9\1\6\3\1\8\9\0 ]] 2023-10-14T00:12:22.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:12:32.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:12:32.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:32.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:32.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:33.457 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:12:33.457 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:12:33.457 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:12:33.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:12:33.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:12:33.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:12:33.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:12:33.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:12:33.459 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:12:33.459 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:12:33.459 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:12:33.459 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:33.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:12:33.649 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:12:33.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:12:33.653 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242353 2023-10-14T00:12:33.653 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:33.653 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/991631890 ]] 2023-10-14T00:12:33.653 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/991631890 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\9\9\1\6\3\1\8\9\0 ]] 2023-10-14T00:12:33.654 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:12:43.655 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:43.656 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:43.656 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:12:43.658 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:44.666 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/991631890 2023-10-14T00:12:44.673 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:12:44.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242364 2023-10-14T00:12:44.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:44.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/991631890 ]] 2023-10-14T00:12:44.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/991631890 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\9\9\1\6\3\1\8\9\0 ]] 2023-10-14T00:12:44.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:12:48.846 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:12:48.846 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:12:48.849 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:12:48.849 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:12:48.849 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:12:48.849 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:12:48.849 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:12:48.850 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:12:48.850 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:12:48.850 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:12:48.850 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:12:48.850 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:12:48.851 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:12:48.851 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:12:48.851 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:12:48.844+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:12:54.676 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:54.678 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:54.680 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:12:54.685 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:55.653 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:12:55.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:12:55.654 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242375 2023-10-14T00:12:55.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:55.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:12:55.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\9\9\1\6\3\1\8\9\0 ]] 2023-10-14T00:12:55.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1901974044 2023-10-14T00:12:57.113 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1901974044 until 2023-10-14T01:12:56.272745+0000 (3600 sec) 2023-10-14T00:12:57.152 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1901974044 2023-10-14T00:12:57.153 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:12:57.838 INFO:tasks.workunit.client.0.smithi081.stderr:listed 6 entries 2023-10-14T00:12:57.869 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:12:57.869 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:13:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.000+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.000+0000 7f3980385640 -1 librbd::ImageState: 0x563ddef17a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.004+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.004+0000 7f3980385640 -1 librbd::ImageState: 0x563ddec68e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.004+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.008+0000 7f3980385640 -1 librbd::ImageState: 0x563ddea1ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.008+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.012+0000 7f3980385640 -1 librbd::ImageState: 0x563ddea1eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.016+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.016+0000 7f3980385640 -1 librbd::ImageState: 0x563ddeb6ac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.024+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.024+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa3500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.032+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.032+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa2f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.036+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.036+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa3100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.044+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.044+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa3700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.052+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.052+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa2d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.056+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.056+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.064+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.068+0000 7f3980385640 -1 librbd::ImageState: 0x563dde049d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.072+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.075 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.072+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddefa3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.080+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.080+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddefa3b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.088+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.088+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddefa3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.108 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.104+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.109 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.104+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.112 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.108+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.112 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.108+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf054180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.126 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.124+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.127 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.124+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf054380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.130 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.128+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.131 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.128+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf054580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.132+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.135 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:13:00.132+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf054780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:07.871 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:07.872 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:07.873 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:13:07.881 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:08.827 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:13:08.827 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:13:08.828 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242388 2023-10-14T00:13:08.828 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:08.829 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:13:08.829 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:13:08.829 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:13:15.346 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:13:15.347 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:13:15.348 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:13:15.348 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:13:15.349 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:13:15.349 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:13:15.349 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:13:15.349 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:13:15.349 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:13:15.350 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:13:15.350 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:13:15.350 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:13:15.350 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:13:15.351 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.348+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:13:15.351 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:13:15.348+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:13:18.833 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:13:18.853 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:18.854 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:18.869 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:19.880 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:13:19.880 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:13:19.880 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242399 2023-10-14T00:13:19.881 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:19.881 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:13:19.881 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:13:19.881 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:13:29.882 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:13:29.883 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:29.884 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:29.913 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:30.844 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:13:30.845 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:13:30.845 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242410 2023-10-14T00:13:30.846 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:30.846 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:13:30.846 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:13:30.846 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:13:40.853 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:40.867 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:13:40.871 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:40.872 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:41.794 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:13:41.795 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:13:41.796 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242421 2023-10-14T00:13:41.796 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:41.796 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:13:41.797 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:13:41.797 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:13:51.801 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:51.801 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:13:51.811 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:51.812 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:52.710 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:13:52.711 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:13:52.711 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242432 2023-10-14T00:13:52.711 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:52.712 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:13:52.712 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:13:52.712 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:14:02.713 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:14:02.713 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:02.721 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:02.732 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:03.578 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1901974044 2023-10-14T00:14:03.579 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:14:03.579 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242443 2023-10-14T00:14:03.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:03.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1901974044 ]] 2023-10-14T00:14:03.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1901974044 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:14:03.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:14:13.591 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:13.596 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:13.598 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:14:13.612 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:14.489 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:14:14.489 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:14:14.490 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242454 2023-10-14T00:14:14.490 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:14.490 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1791769016 ]] 2023-10-14T00:14:14.490 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1791769016 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\0\1\9\7\4\0\4\4 ]] 2023-10-14T00:14:14.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1791769016 2023-10-14T00:14:16.139 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1791769016 until 2023-10-14T01:14:15.395249+0000 (3600 sec) 2023-10-14T00:14:16.169 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:14:16.172 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1791769016 2023-10-14T00:14:16.984 INFO:tasks.workunit.client.0.smithi081.stderr:listed 7 entries 2023-10-14T00:14:17.017 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:14:17.017 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:14:27.018 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:14:27.019 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:27.019 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:27.035 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:27.877 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:14:27.877 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:14:27.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242467 2023-10-14T00:14:27.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:27.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1791769016 ]] 2023-10-14T00:14:27.879 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1791769016 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\9\1\7\6\9\0\1\6 ]] 2023-10-14T00:14:27.879 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:14:37.880 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:14:37.881 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:37.881 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:37.881 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:38.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:14:38.043+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:14:38.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:14:38.043+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:14:38.835 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:14:38.835 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:14:38.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242478 2023-10-14T00:14:38.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:38.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1791769016 ]] 2023-10-14T00:14:38.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1791769016 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\9\1\7\6\9\0\1\6 ]] 2023-10-14T00:14:38.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:14:48.837 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:14:48.838 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:48.838 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:48.840 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:49.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:14:49.814 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:14:49.815 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242489 2023-10-14T00:14:49.815 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:49.816 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1791769016 ]] 2023-10-14T00:14:49.816 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1791769016 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\9\1\7\6\9\0\1\6 ]] 2023-10-14T00:14:49.816 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:14:59.825 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:14:59.826 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:59.828 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:59.828 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:14:59.999+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:14:59.999+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.007+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.007+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.015+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.015+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.027+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.027+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.031+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.031+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.035+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.035+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.039+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.039+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.043+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.043+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.047+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.047+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.051+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.051+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.062 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.059+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.062 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.059+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.063+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.063+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.067+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.067+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.075+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.075+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.079+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.079+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.084 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.083+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.083+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.083+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.083+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.087+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.087+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.091+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.095 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:15:00.091+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.781 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:15:00.781 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:15:00.781 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242500 2023-10-14T00:15:00.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:00.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1791769016 ]] 2023-10-14T00:15:00.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1791769016 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\9\1\7\6\9\0\1\6 ]] 2023-10-14T00:15:00.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:15:10.790 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:15:10.791 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:10.792 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:10.794 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:11.750 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1791769016 2023-10-14T00:15:11.750 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:15:11.751 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242511 2023-10-14T00:15:11.751 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:11.751 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1791769016 ]] 2023-10-14T00:15:11.751 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1791769016 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\9\1\7\6\9\0\1\6 ]] 2023-10-14T00:15:11.752 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:15:21.753 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:15:21.755 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:21.764 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:21.776 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:22.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:15:22.628 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:15:22.629 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242522 2023-10-14T00:15:22.629 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:22.629 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2654634281 ]] 2023-10-14T00:15:22.629 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2654634281 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\9\1\7\6\9\0\1\6 ]] 2023-10-14T00:15:22.630 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2654634281 2023-10-14T00:15:24.126 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2654634281 until 2023-10-14T01:15:23.310686+0000 (3600 sec) 2023-10-14T00:15:24.172 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:15:24.173 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2654634281 2023-10-14T00:15:24.916 INFO:tasks.workunit.client.0.smithi081.stderr:listed 8 entries 2023-10-14T00:15:24.944 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:15:24.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:15:34.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:34.950 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:34.963 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:34.967 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:15:35.989 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:15:35.989 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:15:35.989 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242535 2023-10-14T00:15:35.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:35.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2654634281 ]] 2023-10-14T00:15:35.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2654634281 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\6\5\4\6\3\4\2\8\1 ]] 2023-10-14T00:15:35.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:15:45.993 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:15:45.993 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:45.995 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:45.997 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:46.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:15:46.892 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:15:46.893 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242546 2023-10-14T00:15:46.893 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:46.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2654634281 ]] 2023-10-14T00:15:46.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2654634281 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\6\5\4\6\3\4\2\8\1 ]] 2023-10-14T00:15:46.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:15:55.198 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:15:55.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:15:55.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:15:55.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:15:55.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:15:55.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:15:55.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:15:55.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:15:55.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:15:55.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:15:55.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:15:55.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:15:55.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:15:55.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:15:55.202 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:15:55.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:15:56.900 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:15:56.908 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:56.908 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:56.912 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:57.837 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:15:57.838 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:15:57.844 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242557 2023-10-14T00:15:57.844 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:57.844 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2654634281 ]] 2023-10-14T00:15:57.844 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2654634281 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\6\5\4\6\3\4\2\8\1 ]] 2023-10-14T00:15:57.845 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:16:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.006+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.006+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.006+0000 7f3980385640 -1 librbd::ImageState: 0x563dde063780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.006+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf054880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.010+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.010+0000 7f3980385640 -1 librbd::ImageState: 0x563ddef17b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.014+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.014+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf055480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.026+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.026+0000 7f3980385640 -1 librbd::ImageState: 0x563ddef16b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.030+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.030+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa2480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.034+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.034+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf124d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.042+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.042+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf124100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.054+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.054+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf125300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.074+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.074+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.074+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.074+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf124f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.082+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.082+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddefa3000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.090+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.090+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.102+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.102+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.106+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.106+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.112 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.110+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.112 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.110+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.117 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.118+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.117 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.118+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.122+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.122+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.130 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.130+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.131 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.130+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.135 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.134+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.135 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:16:00.134+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:07.846 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:07.847 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:07.851 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:16:07.852 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:08.731 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:16:08.731 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:16:08.732 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242568 2023-10-14T00:16:08.732 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:08.732 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2654634281 ]] 2023-10-14T00:16:08.733 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2654634281 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\6\5\4\6\3\4\2\8\1 ]] 2023-10-14T00:16:08.733 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:16:18.733 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:16:18.740 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:18.744 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:18.744 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:19.617 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2654634281 2023-10-14T00:16:19.617 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:16:19.618 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242579 2023-10-14T00:16:19.619 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:19.622 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2654634281 ]] 2023-10-14T00:16:19.622 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2654634281 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\6\5\4\6\3\4\2\8\1 ]] 2023-10-14T00:16:19.622 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:16:29.620 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:29.627 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:29.629 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:16:29.632 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:30.505 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:16:30.505 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:16:30.506 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242590 2023-10-14T00:16:30.506 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:30.506 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2097712050 ]] 2023-10-14T00:16:30.507 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2097712050 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\6\5\4\6\3\4\2\8\1 ]] 2023-10-14T00:16:30.507 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2097712050 2023-10-14T00:16:31.333 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:16:31.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:16:31.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:16:31.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:16:31.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:16:31.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:16:31.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:16:31.333+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:16:32.130 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2097712050 until 2023-10-14T01:16:31.387465+0000 (3600 sec) 2023-10-14T00:16:32.173 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:16:32.174 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2097712050 2023-10-14T00:16:32.943 INFO:tasks.workunit.client.0.smithi081.stderr:listed 9 entries 2023-10-14T00:16:32.975 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:16:32.975 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:16:42.980 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:16:42.980 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:42.980 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:43.000 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:43.929 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:16:43.930 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:16:43.930 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242603 2023-10-14T00:16:43.931 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:43.931 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2097712050 ]] 2023-10-14T00:16:43.931 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2097712050 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\9\7\7\1\2\0\5\0 ]] 2023-10-14T00:16:43.931 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:16:53.935 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:16:53.936 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:53.936 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:53.956 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:54.884 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:16:54.885 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:16:54.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242614 2023-10-14T00:16:54.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:54.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2097712050 ]] 2023-10-14T00:16:54.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2097712050 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\9\7\7\1\2\0\5\0 ]] 2023-10-14T00:16:54.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:17:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.005+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.005+0000 7f3981387640 -1 librbd::ImageState: 0x563ddee8f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.009+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.009+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.013+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.013+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf220d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.023 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.021+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.021+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf220f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.033+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.033+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.037+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.037+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.044 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.045+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.045+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.053+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.053+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.053+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.053+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.061+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.061+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.065+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.065+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.069+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.069+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.073+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.073+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.077+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.077+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.081+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.081+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.089+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.089+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.089 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.089+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.089 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.089+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.093+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.093+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.095 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.093+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:17:00.093+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:04.333 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:17:04.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:17:04.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:17:04.334 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:17:04.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:17:04.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:17:04.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:04.329+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:17:04.890 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:17:04.891 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:04.891 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:04.895 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:05.751 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:17:05.751 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:17:05.752 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242625 2023-10-14T00:17:05.752 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:05.752 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2097712050 ]] 2023-10-14T00:17:05.752 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2097712050 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\9\7\7\1\2\0\5\0 ]] 2023-10-14T00:17:05.752 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:17:15.756 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:15.757 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:15.759 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:15.761 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:17:16.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:17:16.580 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:17:16.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242636 2023-10-14T00:17:16.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:16.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2097712050 ]] 2023-10-14T00:17:16.582 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2097712050 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\9\7\7\1\2\0\5\0 ]] 2023-10-14T00:17:16.582 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:17:26.588 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:17:26.590 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:26.593 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:26.593 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:27.422 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2097712050 2023-10-14T00:17:27.423 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:17:27.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242647 2023-10-14T00:17:27.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:27.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2097712050 ]] 2023-10-14T00:17:27.424 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2097712050 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\9\7\7\1\2\0\5\0 ]] 2023-10-14T00:17:27.424 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:17:37.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:17:37.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:37.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:37.447 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:38.430 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:17:38.435 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:17:38.441 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242658 2023-10-14T00:17:38.441 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:38.442 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:17:38.442 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\9\7\7\1\2\0\5\0 ]] 2023-10-14T00:17:38.442 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2598625570 2023-10-14T00:17:40.158 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2598625570 until 2023-10-14T01:17:39.367747+0000 (3600 sec) 2023-10-14T00:17:40.189 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2598625570 2023-10-14T00:17:40.189 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:17:40.937 INFO:tasks.workunit.client.0.smithi081.stderr:listed 10 entries 2023-10-14T00:17:40.968 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:17:40.969 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:17:50.972 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:17:50.973 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:50.973 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:50.984 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:51.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:17:51.882 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:17:51.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242671 2023-10-14T00:17:51.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:51.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:17:51.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:17:51.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:17:54.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:54.084+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:17:54.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:54.084+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:17:54.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:54.084+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:17:54.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:54.084+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:17:56.002 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:56.000+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:17:56.002 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:56.000+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:17:56.003 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:56.000+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:17:56.005 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:56.000+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:17:56.006 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:56.000+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:17:56.006 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:17:56.004+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:18:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.000+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.000+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf221e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.000+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.004+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf124180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.008+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.008+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf055b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.008+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.008+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef16e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.012+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.012+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf054a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.016+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.016+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.016+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.016+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.020+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.020+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.024+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.024+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.028+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.028+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.032+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.032+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.036+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.036+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.040+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.040+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.044+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.044+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.056+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.056+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.060+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.060+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.064+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.064+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.072+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.072+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.080+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.080+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.084+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:18:00.084+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:01.889 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:18:01.890 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:01.891 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:01.894 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:02.816 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:18:02.817 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:18:02.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242682 2023-10-14T00:18:02.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:02.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:18:02.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:18:02.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:18:03.044 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:18:03.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:18:03.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:18:03.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:18:03.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:18:03.046 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:18:03.046 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:18:03.046 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:18:03.047 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.044+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:18:03.051 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.052+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:18:03.052 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:18:03.052+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:18:12.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:12.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:12.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:12.826 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:18:13.760 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:18:13.760 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:18:13.761 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242693 2023-10-14T00:18:13.761 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:13.761 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:18:13.762 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:18:13.762 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:18:23.199 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:18:23.196+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:18:23.763 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:18:23.767 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:23.775 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:23.778 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:24.497 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:18:24.498 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:18:24.499 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242704 2023-10-14T00:18:24.499 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:24.501 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:18:24.501 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:18:24.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:18:34.512 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:34.514 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:34.520 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:18:34.523 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:35.432 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:18:35.433 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:18:35.433 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242715 2023-10-14T00:18:35.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:35.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:18:35.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:18:35.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:18:45.435 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:45.435 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:18:45.438 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:45.443 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:46.343 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2598625570 2023-10-14T00:18:46.343 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:18:46.344 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242726 2023-10-14T00:18:46.344 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:46.344 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2598625570 ]] 2023-10-14T00:18:46.344 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2598625570 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:18:46.345 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:18:51.279 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:18:51.275+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:18:56.346 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:18:56.347 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:56.356 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:56.357 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:57.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:18:57.227 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:18:57.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242737 2023-10-14T00:18:57.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:57.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2020033970 ]] 2023-10-14T00:18:57.229 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2020033970 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\5\9\8\6\2\5\5\7\0 ]] 2023-10-14T00:18:57.229 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2020033970 2023-10-14T00:18:59.176 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2020033970 until 2023-10-14T01:18:58.358010+0000 (3600 sec) 2023-10-14T00:18:59.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:18:59.240 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2020033970 2023-10-14T00:19:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf071900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.007+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.007+0000 7f3981387640 -1 librbd::ImageState: 0x563ddef00300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.011+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.011+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.019+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.019+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ac380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.023+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.023+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ac280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.027+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.029 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.027+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2aca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.031+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.031+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ac900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.035+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.035+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.039+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.039+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ace80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.043+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.043+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2acc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.050 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.047+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.050 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.047+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ac700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.051+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.051+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.059+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.059+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.068 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.067+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.068 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.067+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.071+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.071+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.075+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.075+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ada80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.095+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.100 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.095+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2adc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.103+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.103+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.110 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.107+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.111 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:19:00.107+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ad180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.199 INFO:tasks.workunit.client.0.smithi081.stderr:listed 11 entries 2023-10-14T00:19:00.237 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:19:00.237 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:19:10.245 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:19:10.245 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:10.252 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:10.263 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:11.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:19:11.324 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:19:11.325 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242751 2023-10-14T00:19:11.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:11.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2020033970 ]] 2023-10-14T00:19:11.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2020033970 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\2\0\0\3\3\9\7\0 ]] 2023-10-14T00:19:11.327 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:19:21.331 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:19:21.332 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:21.352 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:21.353 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:22.185 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:19:22.185 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:19:22.185 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242762 2023-10-14T00:19:22.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:22.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2020033970 ]] 2023-10-14T00:19:22.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2020033970 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\2\0\0\3\3\9\7\0 ]] 2023-10-14T00:19:22.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:19:32.190 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:32.197 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:32.198 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:32.204 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:19:33.334 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:19:33.335 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:19:33.335 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242773 2023-10-14T00:19:33.336 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:33.336 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2020033970 ]] 2023-10-14T00:19:33.336 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2020033970 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\2\0\0\3\3\9\7\0 ]] 2023-10-14T00:19:33.336 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:19:43.337 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:19:43.338 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:43.359 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:43.363 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:44.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:19:44.225 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:19:44.226 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242784 2023-10-14T00:19:44.226 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:44.226 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2020033970 ]] 2023-10-14T00:19:44.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2020033970 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\2\0\0\3\3\9\7\0 ]] 2023-10-14T00:19:44.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:19:54.230 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:19:54.230 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:54.238 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:54.244 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:55.116 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2020033970 2023-10-14T00:19:55.116 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:19:55.117 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242795 2023-10-14T00:19:55.117 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:55.118 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2020033970 ]] 2023-10-14T00:19:55.118 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2020033970 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\2\0\0\3\3\9\7\0 ]] 2023-10-14T00:19:55.118 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:19:58.523 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:19:58.524 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:19:58.524 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:19:58.524 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:19:58.524 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:19:58.524 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:19:58.525 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:19:58.525 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:19:58.525 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:19:58.525 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:19:58.525 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:19:58.526 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:19:58.526 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:19:58.526 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:19:58.527 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:19:58.515+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:20:05.119 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:20:05.119 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:05.124 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:05.126 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:05.917 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:20:05.918 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:20:05.921 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242805 2023-10-14T00:20:05.922 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:05.922 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3088214557 ]] 2023-10-14T00:20:05.922 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3088214557 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\2\0\0\3\3\9\7\0 ]] 2023-10-14T00:20:05.922 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3088214557 2023-10-14T00:20:07.327 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:07.327+0000 7f3980385640 -1 librbd::ImageWatcher: 0x563ddf524900 image watch failed: 94823739785216, (107) Transport endpoint is not connected 2023-10-14T00:20:07.327 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:07.327+0000 7f3980385640 -1 librbd::Watcher: 0x563ddf524900 handle_error: handle=94823739785216: (107) Transport endpoint is not connected 2023-10-14T00:20:07.414 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:07.411+0000 7f3980b86640 -1 librbd::watcher::RewatchRequest: 0x563ddead78b0 handle_unwatch client blocklisted 2023-10-14T00:20:07.414 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:07.411+0000 7f3980b86640 -1 librbd::Watcher: 0x563ddf524900 handle_rewatch: client blocklisted 2023-10-14T00:20:07.760 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.759+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:20:07.761 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.759+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:20:07.761 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.759+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:20:07.766 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:20:07.767 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:20:07.767 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:20:07.767 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:20:07.767 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:20:07.767 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:20:07.768 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:20:07.769 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:20:07.769 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:20:07.769 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:20:07.770 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:20:07.763+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:20:08.190 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3088214557 until 2023-10-14T01:20:07.384156+0000 (3600 sec) 2023-10-14T00:20:08.214 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.211+0000 7f3980b86640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.214 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.211+0000 7f3980b86640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x563dde717930 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.214 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.211+0000 7f3980b86640 -1 librbd::io::AioCompletion: 0x563ddf4a4840 fail: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.215 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.215+0000 7f3980b86640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.215 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.215+0000 7f3980b86640 -1 librbd::io::AioCompletion: 0x563dd9c278c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.215 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.215+0000 7f3980b86640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x563ddf3e8e80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.216 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:20:08.215+0000 7f3980b86640 -1 librbd::managed_lock::ReleaseRequest: 0x563ddef85f20 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:08.229 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:20:08.230 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3088214557 2023-10-14T00:20:08.864 INFO:tasks.workunit.client.0.smithi081.stderr:listed 12 entries 2023-10-14T00:20:08.900 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:20:08.901 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:20:18.908 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:18.908 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:20:18.918 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:18.936 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:19.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:20:19.884 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:20:19.884 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242819 2023-10-14T00:20:19.885 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:19.885 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3088214557 ]] 2023-10-14T00:20:19.885 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3088214557 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\8\8\2\1\4\5\5\7 ]] 2023-10-14T00:20:19.885 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:20:29.914 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:20:29.914 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:29.919 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:29.924 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:30.946 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:20:30.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:20:30.947 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242830 2023-10-14T00:20:30.947 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:30.947 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3088214557 ]] 2023-10-14T00:20:30.948 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3088214557 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\8\8\2\1\4\5\5\7 ]] 2023-10-14T00:20:30.948 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:20:40.950 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:40.950 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:20:40.951 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:40.962 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:41.738 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:20:41.739 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:20:41.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242841 2023-10-14T00:20:41.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:41.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3088214557 ]] 2023-10-14T00:20:41.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3088214557 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\8\8\2\1\4\5\5\7 ]] 2023-10-14T00:20:41.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:20:51.747 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:51.748 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:51.751 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:51.753 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:20:52.597 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:20:52.597 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:20:52.598 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242852 2023-10-14T00:20:52.598 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:52.599 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3088214557 ]] 2023-10-14T00:20:52.599 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3088214557 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\8\8\2\1\4\5\5\7 ]] 2023-10-14T00:20:52.599 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:21:02.602 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:02.606 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:02.613 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:02.617 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:21:03.630 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3088214557 2023-10-14T00:21:03.630 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:21:03.631 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242863 2023-10-14T00:21:03.631 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:03.631 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3088214557 ]] 2023-10-14T00:21:03.632 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3088214557 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\8\8\2\1\4\5\5\7 ]] 2023-10-14T00:21:03.632 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:21:13.632 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:21:13.633 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:13.637 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:13.650 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:14.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:21:14.449 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:21:14.457 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242874 2023-10-14T00:21:14.458 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:14.458 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1782050421 ]] 2023-10-14T00:21:14.458 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1782050421 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\8\8\2\1\4\5\5\7 ]] 2023-10-14T00:21:14.458 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1782050421 2023-10-14T00:21:16.803 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1782050421 until 2023-10-14T01:21:16.095094+0000 (3600 sec) 2023-10-14T00:21:16.832 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:21:16.833 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1782050421 2023-10-14T00:21:17.617 INFO:tasks.workunit.client.0.smithi081.stderr:listed 13 entries 2023-10-14T00:21:17.669 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:21:17.669 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:21:27.672 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:27.673 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:27.673 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:27.674 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:21:28.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:21:28.674 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:21:28.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242888 2023-10-14T00:21:28.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:28.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1782050421 ]] 2023-10-14T00:21:28.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1782050421 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\8\2\0\5\0\4\2\1 ]] 2023-10-14T00:21:28.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:21:38.677 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:38.677 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:38.682 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:38.697 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:21:39.627 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:21:39.627 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:21:39.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242899 2023-10-14T00:21:39.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:39.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1782050421 ]] 2023-10-14T00:21:39.629 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1782050421 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\8\2\0\5\0\4\2\1 ]] 2023-10-14T00:21:39.629 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:21:49.638 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:49.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:21:49.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:49.660 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:50.474 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:21:50.485 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:21:50.486 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242910 2023-10-14T00:21:50.486 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:50.486 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1782050421 ]] 2023-10-14T00:21:50.487 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1782050421 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\8\2\0\5\0\4\2\1 ]] 2023-10-14T00:21:50.487 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:22:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde063f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf2aca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.005+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.005+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf2adf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.009+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.009+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.009+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.009+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.013+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.013+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.017+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.017+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.021+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.021+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.021+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.021+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.025+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.025+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.029+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.029+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.033+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.033+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf305180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.037+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.037+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf305380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.041+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.040 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.041+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf305580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.045+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.045+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf305780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.045+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.045+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf305980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.049+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.049+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf305b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.053+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.053+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf305400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.057+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.057+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf305180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.061+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:22:00.061+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.494 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:22:00.497 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:00.500 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:00.526 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:01.390 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:22:01.392 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:22:01.394 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242921 2023-10-14T00:22:01.394 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:01.394 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1782050421 ]] 2023-10-14T00:22:01.394 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1782050421 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\8\2\0\5\0\4\2\1 ]] 2023-10-14T00:22:01.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:22:11.019 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.009+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:22:11.019 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.009+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:22:11.020 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.009+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:22:11.020 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.009+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:22:11.020 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.009+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:22:11.020 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.013+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:22:11.020 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.013+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:22:11.021 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.013+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:22:11.021 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.013+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:22:11.021 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:22:11.017+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:22:11.395 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:11.396 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:11.397 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:11.399 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:22:12.407 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1782050421 2023-10-14T00:22:12.407 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:22:12.407 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242932 2023-10-14T00:22:12.408 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:12.408 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1782050421 ]] 2023-10-14T00:22:12.408 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1782050421 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\8\2\0\5\0\4\2\1 ]] 2023-10-14T00:22:12.408 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:22:22.410 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:22:22.411 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:22.412 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:22.424 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:23.274 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:22:23.274 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:22:23.275 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242943 2023-10-14T00:22:23.275 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:23.276 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2822357953 ]] 2023-10-14T00:22:23.276 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2822357953 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\7\8\2\0\5\0\4\2\1 ]] 2023-10-14T00:22:23.276 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2822357953 2023-10-14T00:22:25.804 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2822357953 until 2023-10-14T01:22:25.095084+0000 (3600 sec) 2023-10-14T00:22:25.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2822357953 2023-10-14T00:22:25.841 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:22:26.653 INFO:tasks.workunit.client.0.smithi081.stderr:listed 14 entries 2023-10-14T00:22:26.708 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:22:26.708 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:22:36.709 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:22:36.710 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:36.710 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:36.713 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:37.746 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:22:37.746 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:22:37.754 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242957 2023-10-14T00:22:37.754 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:37.754 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2822357953 ]] 2023-10-14T00:22:37.754 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2822357953 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\2\2\3\5\7\9\5\3 ]] 2023-10-14T00:22:37.755 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:22:44.995 INFO:tasks.ceph.osd.2.smithi081.stderr:2023-10-14T00:22:44.997+0000 7fa0ac5d3640 -1 reset not still connected to 0x55e37943d520 2023-10-14T00:22:47.846 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:47.847 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:47.847 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:47.856 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:22:48.992 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:22:48.993 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:22:49.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242969 2023-10-14T00:22:49.003 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:49.003 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2822357953 ]] 2023-10-14T00:22:49.003 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2822357953 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\2\2\3\5\7\9\5\3 ]] 2023-10-14T00:22:49.003 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:22:59.016 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:59.016 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:59.019 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:22:59.019 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:59.877 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:22:59.877 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:22:59.881 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242979 2023-10-14T00:22:59.881 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:59.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2822357953 ]] 2023-10-14T00:22:59.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2822357953 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\2\2\3\5\7\9\5\3 ]] 2023-10-14T00:22:59.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:23:00.000 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.001+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.000 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.001+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf293e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.005+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.005+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf293980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.009+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.009+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf24f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.025+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.025+0000 7f3980385640 -1 librbd::ImageState: 0x563ddec68500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.029+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.029+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf055c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.033+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.033+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf071080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.041+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.041+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf125e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.045+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.045+0000 7f3980385640 -1 librbd::ImageState: 0x563ddefa3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.044 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.045+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.045+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf304c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.049+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.049+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf304e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.053+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.053+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf305200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.061+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.061+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf070c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.065+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.065+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf305580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.069+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.069+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf305780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.073+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.073+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf305980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.077+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.077+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf305b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.081+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.081+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf4da000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.089+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.089+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf4da200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.093+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.093+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf4da400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.097+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:23:00.097+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf4da600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:09.889 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:23:09.893 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:09.900 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:09.908 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:10.833 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:23:10.833 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:23:10.834 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697242990 2023-10-14T00:23:10.834 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:10.835 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2822357953 ]] 2023-10-14T00:23:10.835 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2822357953 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\2\2\3\5\7\9\5\3 ]] 2023-10-14T00:23:10.835 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:23:20.836 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:23:20.837 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:20.837 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:20.854 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:21.773 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2822357953 2023-10-14T00:23:21.773 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:23:21.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243001 2023-10-14T00:23:21.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:21.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2822357953 ]] 2023-10-14T00:23:21.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2822357953 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\2\2\3\5\7\9\5\3 ]] 2023-10-14T00:23:21.775 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:23:21.799 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:23:21.800+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:23:21.799 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:23:21.800+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:23:31.777 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:31.785 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:23:31.791 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:31.792 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:32.917 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:23:32.918 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:23:32.918 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243012 2023-10-14T00:23:32.918 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:32.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3439709489 ]] 2023-10-14T00:23:32.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3439709489 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\2\2\3\5\7\9\5\3 ]] 2023-10-14T00:23:32.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3439709489 2023-10-14T00:23:34.853 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3439709489 until 2023-10-14T01:23:34.065283+0000 (3600 sec) 2023-10-14T00:23:34.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3439709489 2023-10-14T00:23:34.901 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:23:35.720 INFO:tasks.workunit.client.0.smithi081.stderr:listed 15 entries 2023-10-14T00:23:35.804 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:23:35.804 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:23:45.805 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:45.808 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:23:45.813 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:45.825 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:46.684 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:23:46.689 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:23:46.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243026 2023-10-14T00:23:46.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:46.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3439709489 ]] 2023-10-14T00:23:46.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3439709489 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\3\9\7\0\9\4\8\9 ]] 2023-10-14T00:23:46.692 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:23:52.035 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:23:52.032+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:23:52.035 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:23:52.032+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:23:52.035 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:23:52.032+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:23:55.757 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:23:55.756+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:23:56.693 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:23:56.693 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:56.694 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:56.694 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:57.530 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:23:57.530 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:23:57.531 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243037 2023-10-14T00:23:57.531 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:57.531 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3439709489 ]] 2023-10-14T00:23:57.531 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3439709489 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\3\9\7\0\9\4\8\9 ]] 2023-10-14T00:23:57.532 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:24:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.004+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.004+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddefa2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.004+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.004+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf070000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.008+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.008+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf221780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.012+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.012+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde1aa080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.024+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.024+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf055d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.028+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.028+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddee8f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.036 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.036+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.036 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.036+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.044+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.044+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.044+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.044+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4da180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.063 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.064+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.064+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4da900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.076+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.076+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.080+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.080+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.084+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.084+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.089 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.092+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.092+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.100+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.100+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.100+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.100+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.103 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.104+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.103 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.104+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.117 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.116+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.117 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.116+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dbc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.124+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.124+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dbe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.124+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:24:00.128+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:06.002 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:06.004+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:24:06.002 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:06.004+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:24:06.002 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:06.004+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:24:07.546 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:07.547 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:07.548 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:07.554 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:24:08.591 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:24:08.591 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:24:08.593 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243048 2023-10-14T00:24:08.593 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:08.593 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3439709489 ]] 2023-10-14T00:24:08.593 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3439709489 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\3\9\7\0\9\4\8\9 ]] 2023-10-14T00:24:08.594 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:24:18.596 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:24:18.601 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:18.607 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:18.615 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:19.424 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:24:19.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:24:19.426 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243059 2023-10-14T00:24:19.426 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:19.426 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3439709489 ]] 2023-10-14T00:24:19.426 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3439709489 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\3\9\7\0\9\4\8\9 ]] 2023-10-14T00:24:19.427 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:24:26.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:24:26.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:24:26.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:24:26.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:24:26.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:24:26.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:24:26.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:24:26.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:24:26.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:24:26.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:24:26.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:24:26.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:24:26.253 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:24:26.253 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:24:26.253 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:26.243+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:24:29.434 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:29.434 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:29.435 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:24:29.435 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:30.376 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3439709489 2023-10-14T00:24:30.376 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:24:30.376 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243070 2023-10-14T00:24:30.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:30.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3439709489 ]] 2023-10-14T00:24:30.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3439709489 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\3\9\7\0\9\4\8\9 ]] 2023-10-14T00:24:30.377 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:24:36.844 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:24:36.845 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:24:36.845 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:24:36.845 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:24:36.845 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:24:36.845 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:24:36.846 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:24:36.846 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:24:36.846 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:24:36.846 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:36.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:24:38.289 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:38.287+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:24:38.289 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:38.287+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:24:40.378 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:24:40.378 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:40.383 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:40.394 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:41.351 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:24:41.351 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:24:41.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243081 2023-10-14T00:24:41.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:41.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1915003886 ]] 2023-10-14T00:24:41.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1915003886 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\4\3\9\7\0\9\4\8\9 ]] 2023-10-14T00:24:41.354 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1915003886 2023-10-14T00:24:43.288 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1915003886 until 2023-10-14T01:24:42.532353+0000 (3600 sec) 2023-10-14T00:24:43.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:24:43.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1915003886 2023-10-14T00:24:44.165 INFO:tasks.workunit.client.0.smithi081.stderr:listed 16 entries 2023-10-14T00:24:44.206 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:24:44.207 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:24:54.208 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:24:54.209 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:54.227 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:54.229 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:55.727 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:24:55.729 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:24:55.730 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243095 2023-10-14T00:24:55.730 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:55.731 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1915003886 ]] 2023-10-14T00:24:55.731 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1915003886 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\1\5\0\0\3\8\8\6 ]] 2023-10-14T00:24:55.731 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:24:55.797 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:24:55.800 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:24:55.800 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:24:55.800 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:24:55.800 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:24:55.800 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:24:55.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:24:55.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:24:55.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:25:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.003+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.003+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf2ac800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.003+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.003+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf071b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.011+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.011+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf221300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.019+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.019+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf070e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.027+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.027+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf293700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.035+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.035+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.047+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.047+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.055+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.055+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.063 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.063+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.063 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.063+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.067+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.067+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.075+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.075+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.083+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.083+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.083+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.085 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.083+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.095+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.095+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.103+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.103+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.114 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.115+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.114 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.115+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.123+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.123+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.123+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.123+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.128 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.127+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.128 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.127+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.132 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.131+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.132 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:25:00.131+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:05.734 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:25:05.734 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:05.734 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:05.743 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:06.720 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:25:06.767 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:25:06.768 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243106 2023-10-14T00:25:06.768 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:06.768 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1915003886 ]] 2023-10-14T00:25:06.769 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1915003886 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\1\5\0\0\3\8\8\6 ]] 2023-10-14T00:25:06.769 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:25:16.722 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:16.725 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:16.725 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:16.725 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:25:17.617 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:25:17.617 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:25:17.618 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243117 2023-10-14T00:25:17.618 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:17.619 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1915003886 ]] 2023-10-14T00:25:17.619 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1915003886 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\1\5\0\0\3\8\8\6 ]] 2023-10-14T00:25:17.619 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:25:27.626 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:25:27.629 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:27.647 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:27.650 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:28.460 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:25:28.460 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:25:28.461 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243128 2023-10-14T00:25:28.461 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:28.461 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1915003886 ]] 2023-10-14T00:25:28.462 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1915003886 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\1\5\0\0\3\8\8\6 ]] 2023-10-14T00:25:28.462 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:25:38.474 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:25:38.475 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:38.479 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:38.484 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:39.391 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1915003886 2023-10-14T00:25:39.392 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:25:39.392 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243139 2023-10-14T00:25:39.392 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:39.392 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1915003886 ]] 2023-10-14T00:25:39.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1915003886 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\1\5\0\0\3\8\8\6 ]] 2023-10-14T00:25:39.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:25:49.414 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:49.415 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:49.415 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:49.417 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:25:50.344 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:25:50.354 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:25:50.355 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243150 2023-10-14T00:25:50.355 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:50.355 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:25:50.356 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\9\1\5\0\0\3\8\8\6 ]] 2023-10-14T00:25:50.356 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3249982660 2023-10-14T00:25:52.881 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3249982660 until 2023-10-14T01:25:52.078504+0000 (3600 sec) 2023-10-14T00:25:52.929 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:25:52.935 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3249982660 2023-10-14T00:25:53.668 INFO:tasks.workunit.client.0.smithi081.stderr:listed 17 entries 2023-10-14T00:25:53.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:25:53.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:26:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.005+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.005+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ac980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.005+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.005+0000 7f3981387640 -1 librbd::ImageState: 0x563ddefa2680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.013+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.013+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.021+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.021+0000 7f3981387640 -1 librbd::ImageState: 0x563ddee8ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.033+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.033+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.041+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.041+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf125600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.049+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.049+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf125100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.069+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.069+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf4dae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.081+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.081+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf070400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.085+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.085+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf220900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.087 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.089+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.089+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf220300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.092 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.093+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.092 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.093+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf221700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.101 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.101+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.101 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.101+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.109+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.109+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf292700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.107 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.109+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.107 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.109+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf292400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.110 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.113+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.110 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.113+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf323800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.113 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.113+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.113 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.113+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf323500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.117+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.117+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf322800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.130 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.133+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.131 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.133+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.137+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:26:00.137+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf054e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:03.699 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:26:03.699 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:03.705 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:03.713 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:04.590 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:26:04.591 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:26:04.591 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243164 2023-10-14T00:26:04.591 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:04.592 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:26:04.592 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:26:04.592 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:26:14.593 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:26:14.594 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:14.594 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:14.603 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:15.546 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:26:15.546 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:26:15.547 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243175 2023-10-14T00:26:15.547 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:15.547 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:26:15.548 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:26:15.548 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:26:25.559 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:26:25.559 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:25.565 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:25.571 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:26.401 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:26:26.401 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:26:26.402 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243186 2023-10-14T00:26:26.402 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:26.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:26:26.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:26:26.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:26:27.041 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:26:27.041+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:26:36.404 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:26:36.404 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:36.405 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:36.417 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:37.356 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:26:37.356 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:26:37.357 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243197 2023-10-14T00:26:37.357 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:37.357 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:26:37.357 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:26:37.358 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:26:47.359 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:47.361 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:47.361 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:47.380 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:26:48.280 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:26:48.280 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:26:48.281 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243208 2023-10-14T00:26:48.281 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:48.281 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:26:48.282 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:26:48.282 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:26:58.287 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:58.288 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:58.289 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:26:58.305 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:59.109 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3249982660 2023-10-14T00:26:59.109 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:26:59.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243219 2023-10-14T00:26:59.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:59.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3249982660 ]] 2023-10-14T00:26:59.111 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3249982660 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:26:59.111 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:27:09.125 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:27:09.125 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:09.126 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:09.129 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:09.273 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:27:09.274 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:27:09.274 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:27:09.274 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:27:09.275 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:27:09.275 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:27:09.275 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:27:09.275 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:27:09.275 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:27:09.276 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:27:09.276 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:09.276+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:27:10.145 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:27:10.145 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:27:10.146 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243230 2023-10-14T00:27:10.146 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:10.146 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2014696063 ]] 2023-10-14T00:27:10.147 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2014696063 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\9\9\8\2\6\6\0 ]] 2023-10-14T00:27:10.147 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2014696063 2023-10-14T00:27:12.346 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2014696063 until 2023-10-14T01:27:11.553081+0000 (3600 sec) 2023-10-14T00:27:12.390 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2014696063 2023-10-14T00:27:12.391 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:27:13.092 INFO:tasks.workunit.client.0.smithi081.stderr:listed 18 entries 2023-10-14T00:27:13.116 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:27:13.117 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:27:17.082 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:27:17.083 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:27:17.083 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:27:17.083 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:27:17.083 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:27:17.083 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:27:17.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:27:17.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:27:17.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:27:17.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:27:17.084 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:27:17.085 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:27:17.085 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:27:17.085 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:27:17.085 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:17.080+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:27:23.118 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:27:23.120 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:23.123 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:23.124 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:23.996 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:27:23.997 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:27:23.998 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243244 2023-10-14T00:27:23.998 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:23.999 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2014696063 ]] 2023-10-14T00:27:23.999 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2014696063 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\1\4\6\9\6\0\6\3 ]] 2023-10-14T00:27:23.999 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:27:33.999 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:27:34.000 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:34.000 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:34.000 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:34.817 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:27:34.817 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:27:34.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243254 2023-10-14T00:27:34.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:34.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2014696063 ]] 2023-10-14T00:27:34.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2014696063 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\1\4\6\9\6\0\6\3 ]] 2023-10-14T00:27:34.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:27:42.202 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:27:42.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:27:42.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:27:42.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:27:42.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:27:42.204 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:27:42.204 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:27:42.204 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:27:42.204 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:27:42.204 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:27:42.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:27:42.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:27:42.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:27:42.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:27:42.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:27:42.204+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:27:44.826 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:27:44.826 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:44.842 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:44.849 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:45.989 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:27:45.989 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:27:45.989 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243265 2023-10-14T00:27:45.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:45.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2014696063 ]] 2023-10-14T00:27:45.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2014696063 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\1\4\6\9\6\0\6\3 ]] 2023-10-14T00:27:45.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:27:55.991 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:27:55.992 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:55.992 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:55.992 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:56.925 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:27:56.925 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:27:56.926 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243276 2023-10-14T00:27:56.927 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:56.927 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2014696063 ]] 2023-10-14T00:27:56.927 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2014696063 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\1\4\6\9\6\0\6\3 ]] 2023-10-14T00:27:56.928 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:28:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:27:59.999+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:27:59.999+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24f400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.003+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.003+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf293500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.007+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.007+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf125900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.011+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.011+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf292d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.015+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.019+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.023+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.023+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.027+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.027+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.035+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.035+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf071080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.039+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.039+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf221a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.047+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.047+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf221580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.051+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.051+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf322c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.059+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.059+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.062 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.063+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.063 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.063+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b4c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.067+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.067+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.071+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.071+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.075+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.079+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b4a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.083+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.083 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.083+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b4800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.087+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.087+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.091+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.091+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b5200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.099+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:00.099+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:06.932 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:06.936 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:06.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:06.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:28:08.055 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2014696063 2023-10-14T00:28:08.055 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:28:08.056 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243288 2023-10-14T00:28:08.056 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:08.056 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2014696063 ]] 2023-10-14T00:28:08.056 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2014696063 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\1\4\6\9\6\0\6\3 ]] 2023-10-14T00:28:08.057 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:28:18.057 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:28:18.057 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:18.065 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:18.071 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:19.043 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:28:19.044 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:28:19.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243299 2023-10-14T00:28:19.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3142402495 ]] 2023-10-14T00:28:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3142402495 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\0\1\4\6\9\6\0\6\3 ]] 2023-10-14T00:28:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3142402495 2023-10-14T00:28:20.925 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3142402495 until 2023-10-14T01:28:20.128222+0000 (3600 sec) 2023-10-14T00:28:20.974 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3142402495 2023-10-14T00:28:20.976 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:28:21.845 INFO:tasks.workunit.client.0.smithi081.stderr:listed 19 entries 2023-10-14T00:28:21.872 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:28:21.872 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:28:30.429 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:28:30.423+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:28:30.436 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:28:30.435+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:28:30.436 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:28:30.435+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:28:30.436 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:28:30.435+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:28:30.436 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:28:30.435+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:28:31.880 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:28:31.881 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:31.890 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:31.890 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:32.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:28:32.784 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:28:32.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243312 2023-10-14T00:28:32.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:32.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3142402495 ]] 2023-10-14T00:28:32.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3142402495 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\4\2\4\0\2\4\9\5 ]] 2023-10-14T00:28:32.786 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:28:42.787 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:42.787 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:42.795 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:42.795 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:28:43.647 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:28:43.648 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:28:43.649 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243323 2023-10-14T00:28:43.649 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:43.649 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3142402495 ]] 2023-10-14T00:28:43.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3142402495 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\4\2\4\0\2\4\9\5 ]] 2023-10-14T00:28:43.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:28:53.665 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:53.666 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:53.668 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:28:53.685 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:54.464 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:28:54.469 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:28:54.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243334 2023-10-14T00:28:54.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:54.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3142402495 ]] 2023-10-14T00:28:54.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3142402495 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\4\2\4\0\2\4\9\5 ]] 2023-10-14T00:28:54.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:29:00.000 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:59.998+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.000 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:28:59.998+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf2ac000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.002+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.002+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.006+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.006+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf124d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.014+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.014+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf305680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.022+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.022+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf323680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.025 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.026+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.026+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b4e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.038+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.038+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.050+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.050+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b4f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.054+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.054+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.082+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.082+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.086+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.087 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.086+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.094+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.094+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b4280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.100 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.098+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.101 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.098+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.102+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.102+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.108 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.106+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.108 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.106+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b5f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.122+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.122+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.130+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.130+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.134+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.134+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.138+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.138 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.138+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf66c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.140 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.138+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.140 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:29:00.138+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:04.472 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:04.475 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:04.481 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:04.496 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:29:05.237 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:29:05.238 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:29:05.238 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243345 2023-10-14T00:29:05.238 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:05.239 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3142402495 ]] 2023-10-14T00:29:05.239 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3142402495 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\4\2\4\0\2\4\9\5 ]] 2023-10-14T00:29:05.239 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:29:15.241 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:29:15.242 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:15.242 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:15.249 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:16.178 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3142402495 2023-10-14T00:29:16.191 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:29:16.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243356 2023-10-14T00:29:16.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:16.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3142402495 ]] 2023-10-14T00:29:16.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3142402495 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\4\2\4\0\2\4\9\5 ]] 2023-10-14T00:29:16.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:29:26.197 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:26.203 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:26.203 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:29:26.204 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:27.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:29:27.085 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:29:27.086 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243367 2023-10-14T00:29:27.086 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:27.086 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3583908646 ]] 2023-10-14T00:29:27.086 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3583908646 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\4\2\4\0\2\4\9\5 ]] 2023-10-14T00:29:27.087 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3583908646 2023-10-14T00:29:28.930 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3583908646 until 2023-10-14T01:29:28.181128+0000 (3600 sec) 2023-10-14T00:29:28.986 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:29:28.995 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3583908646 2023-10-14T00:29:29.847 INFO:tasks.workunit.client.0.smithi081.stderr:listed 20 entries 2023-10-14T00:29:29.872 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:29:29.872 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:29:39.874 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:39.880 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:39.881 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:29:39.885 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:40.714 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:29:40.717 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:29:40.721 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243380 2023-10-14T00:29:40.722 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:40.722 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3583908646 ]] 2023-10-14T00:29:40.722 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3583908646 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\5\8\3\9\0\8\6\4\6 ]] 2023-10-14T00:29:40.722 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:29:43.219 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:29:43.222+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:29:50.726 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:50.727 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:50.733 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:29:50.739 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:51.619 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:29:51.626 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:29:51.627 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243391 2023-10-14T00:29:51.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:51.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3583908646 ]] 2023-10-14T00:29:51.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3583908646 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\5\8\3\9\0\8\6\4\6 ]] 2023-10-14T00:29:51.628 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:29:57.806 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:29:57.809+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:29:57.810 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:29:57.811 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:29:57.811 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:29:57.811 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:29:57.811 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:29:57.811 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:29:57.812 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:29:57.812 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:29:57.812 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:29:57.812 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:29:57.812 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:29:57.813 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:29:57.813 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:29:57.813 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:29:57.813 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:57.813+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:29:58.187 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.189+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:29:58.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:29:58.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:29:58.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:29:58.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:29:58.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:29:58.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:29:58.201 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:29:58.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:30:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.005+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.005+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf2ad380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.009+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.009+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.013+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.013+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.021+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.021+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.021+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.021+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.037+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.037+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.049+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.049+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.057+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.057+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.057+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.057+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.061+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.061+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.081+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.081+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.085+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.082 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.085+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.089+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.089+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.097+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.097+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.097+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.097+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.117+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.117 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.117+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.123 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.125+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.123 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.125+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.133+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.133+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66cc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.137+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.137+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.137+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:00.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:30:00.137+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66dd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:30:01.656 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:01.656 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:30:01.657 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:01.669 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:02.473 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:30:02.473 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:30:02.478 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243402 2023-10-14T00:30:02.478 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:02.478 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3583908646 ]] 2023-10-14T00:30:02.478 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3583908646 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\5\8\3\9\0\8\6\4\6 ]] 2023-10-14T00:30:02.479 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:30:12.489 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:30:12.492 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:12.497 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:12.507 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:13.489 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:30:13.490 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:30:13.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243413 2023-10-14T00:30:13.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:13.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3583908646 ]] 2023-10-14T00:30:13.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3583908646 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\5\8\3\9\0\8\6\4\6 ]] 2023-10-14T00:30:13.492 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:30:23.506 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:30:23.508 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:23.509 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:23.535 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:24.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3583908646 2023-10-14T00:30:24.395 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:30:24.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243424 2023-10-14T00:30:24.396 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:24.396 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3583908646 ]] 2023-10-14T00:30:24.396 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3583908646 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\5\8\3\9\0\8\6\4\6 ]] 2023-10-14T00:30:24.396 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:30:34.398 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:30:34.404 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:34.407 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:34.410 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:35.307 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:30:35.307 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:30:35.308 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243435 2023-10-14T00:30:35.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:35.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/805000911 ]] 2023-10-14T00:30:35.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/805000911 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\5\8\3\9\0\8\6\4\6 ]] 2023-10-14T00:30:35.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/805000911 2023-10-14T00:30:37.937 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/805000911 until 2023-10-14T01:30:37.165701+0000 (3600 sec) 2023-10-14T00:30:37.982 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:30:37.982 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/805000911 2023-10-14T00:30:38.721 INFO:tasks.workunit.client.0.smithi081.stderr:listed 21 entries 2023-10-14T00:30:38.745 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:30:38.745 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:30:48.755 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:48.755 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:48.756 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:48.761 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:30:49.642 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:30:49.643 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:30:49.643 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243449 2023-10-14T00:30:49.644 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:49.644 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/805000911 ]] 2023-10-14T00:30:49.644 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/805000911 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\0\5\0\0\0\9\1\1 ]] 2023-10-14T00:30:49.644 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:30:59.648 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:30:59.648 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:59.649 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:59.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.017+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.017+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf2adc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.021+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.021+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.033+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.032 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.033+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4dba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.049+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.049+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde062480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.061+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.061+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.061+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.061+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.069+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.069+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.073+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.073+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.077+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.077+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.097+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.097+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.097+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.097+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.097+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.097+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf305580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.125+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.122 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.125+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.137+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.137+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.136 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.137+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.136 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.141+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.139 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.141+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.139 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.141+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66d780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.142 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.145+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.142 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.145+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.145 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.149+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.146 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.149+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.148 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.149+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.149 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.149+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.151 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.153+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.151 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:31:00.153+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.671 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:31:00.671 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:31:00.671 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243460 2023-10-14T00:31:00.672 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:00.672 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/805000911 ]] 2023-10-14T00:31:00.672 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/805000911 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\0\5\0\0\0\9\1\1 ]] 2023-10-14T00:31:00.673 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:31:10.689 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:31:10.692 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:10.692 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:10.705 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:11.486 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:31:11.487 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:31:11.487 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243471 2023-10-14T00:31:11.487 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:11.488 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/805000911 ]] 2023-10-14T00:31:11.488 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/805000911 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\0\5\0\0\0\9\1\1 ]] 2023-10-14T00:31:11.488 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:31:21.488 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:21.492 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:21.494 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:31:21.511 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:22.394 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:31:22.394 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:31:22.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243482 2023-10-14T00:31:22.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:22.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/805000911 ]] 2023-10-14T00:31:22.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/805000911 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\0\5\0\0\0\9\1\1 ]] 2023-10-14T00:31:22.396 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:31:32.397 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:31:32.398 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:32.403 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:32.410 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:33.346 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/805000911 2023-10-14T00:31:33.346 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:31:33.346 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243493 2023-10-14T00:31:33.347 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:33.347 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/805000911 ]] 2023-10-14T00:31:33.347 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/805000911 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\0\5\0\0\0\9\1\1 ]] 2023-10-14T00:31:33.347 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:31:43.348 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:31:43.349 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:43.353 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:43.356 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:44.256 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:31:44.256 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:31:44.258 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243504 2023-10-14T00:31:44.258 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:44.258 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:31:44.258 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\0\5\0\0\0\9\1\1 ]] 2023-10-14T00:31:44.258 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1063646072 2023-10-14T00:31:46.436 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1063646072 until 2023-10-14T01:31:45.634502+0000 (3600 sec) 2023-10-14T00:31:46.474 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:31:46.475 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1063646072 2023-10-14T00:31:47.239 INFO:tasks.workunit.client.0.smithi081.stderr:listed 22 entries 2023-10-14T00:31:47.291 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:31:47.291 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:31:57.313 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:57.313 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:57.314 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:57.315 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:31:58.268 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:31:58.268 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:31:58.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243518 2023-10-14T00:31:58.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:58.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:31:58.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:31:58.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:32:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.004+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.004+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde1aa280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.004+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.004+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.008+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.008+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b4c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.012+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.012+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.016+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.016+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b4980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.028+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.028+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf054180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.029 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.032+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.030 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.032+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64e800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.036+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.036+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.040+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.040+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.044+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.044+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.052+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.052+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.052+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.052+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.056+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.056 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.056+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.060+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.060+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.072+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.072+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.076+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.076+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.100+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.100+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.104+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.104+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.105 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.108+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.105 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.108+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.109 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.112+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.109 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:32:00.112+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:02.038 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:32:02.040+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:32:08.271 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:08.274 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:08.274 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:32:08.275 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:09.138 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:32:09.143 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:32:09.150 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243529 2023-10-14T00:32:09.150 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:09.150 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:32:09.150 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:32:09.151 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:32:12.731 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:12.732+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:32:12.732 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:12.732+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:32:12.733 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:12.732+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:32:12.733 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:12.732+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:32:19.153 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:19.153 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:32:19.155 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:19.170 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:20.183 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:32:20.184 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:32:20.184 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243540 2023-10-14T00:32:20.184 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:20.184 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:32:20.185 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:32:20.185 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:32:30.186 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:32:30.186 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:30.188 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:30.192 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:31.130 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:32:31.130 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:32:31.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243551 2023-10-14T00:32:31.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:31.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:32:31.132 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:32:31.132 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:32:41.133 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:32:41.134 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:41.135 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:41.143 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:42.020 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:32:42.020 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:32:42.021 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243562 2023-10-14T00:32:42.022 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:42.022 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:32:42.022 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:32:42.023 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:32:51.363 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:51.363+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:32:51.363 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:51.363+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:32:51.363 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:51.363+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:32:51.364 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:51.363+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:32:51.364 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:51.363+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:32:51.364 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:51.363+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:32:52.024 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:32:52.028 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:52.043 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:52.044 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:52.923 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1063646072 2023-10-14T00:32:52.923 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:32:52.924 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243572 2023-10-14T00:32:52.924 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:52.924 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1063646072 ]] 2023-10-14T00:32:52.924 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1063646072 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:32:52.925 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:32:58.949 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:32:58.947+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:33:01.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:01.207+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:33:01.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:01.207+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:33:02.926 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:02.926 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:02.926 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:33:02.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:03.773 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:33:03.773 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:33:03.773 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243583 2023-10-14T00:33:03.773 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:03.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/60711583 ]] 2023-10-14T00:33:03.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/60711583 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\6\3\6\4\6\0\7\2 ]] 2023-10-14T00:33:03.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/60711583 2023-10-14T00:33:05.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:33:05.094+0000 7f3981387640 -1 librbd::managed_lock::GetLockerRequest: 0x563dde94f180 handle_get_lockers: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:05.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:33:05.094+0000 7f3981387640 -1 librbd::managed_lock::AcquireRequest: 0x563ddf896000 handle_get_locker: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:05.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:33:05.094+0000 7f3981387640 -1 librbd::ManagedLock: 0x563dde1a11f8 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:05.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:33:05.094+0000 7f3981387640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x563dde717ad0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:05.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:33:05.094+0000 7f3981387640 -1 librbd::io::AioCompletion: 0x563ddf44a2c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:05.990 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/60711583 until 2023-10-14T01:33:05.193566+0000 (3600 sec) 2023-10-14T00:33:06.019 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:33:06.019 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/60711583 2023-10-14T00:33:06.683 INFO:tasks.workunit.client.0.smithi081.stderr:listed 23 entries 2023-10-14T00:33:06.713 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:33:06.713 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:33:15.555 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:15.554+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:33:15.555 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:15.554+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:33:16.715 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:33:16.715 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:16.715 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:16.728 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:17.589 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:33:17.589 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:33:17.590 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243597 2023-10-14T00:33:17.590 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:17.591 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/60711583 ]] 2023-10-14T00:33:17.591 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/60711583 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\6\0\7\1\1\5\8\3 ]] 2023-10-14T00:33:17.591 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:33:17.765 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:17.770+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:33:17.765 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:17.770+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:33:17.765 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:17.770+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:33:17.766 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:17.770+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:33:27.598 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:33:27.599 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:27.600 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:27.603 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:28.436 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:33:28.436 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:33:28.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243608 2023-10-14T00:33:28.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:28.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/60711583 ]] 2023-10-14T00:33:28.438 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/60711583 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\6\0\7\1\1\5\8\3 ]] 2023-10-14T00:33:28.438 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:33:38.439 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:33:38.439 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:38.454 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:38.462 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:39.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:33:39.310 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:33:39.311 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243619 2023-10-14T00:33:39.311 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:39.311 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/60711583 ]] 2023-10-14T00:33:39.311 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/60711583 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\6\0\7\1\1\5\8\3 ]] 2023-10-14T00:33:39.312 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:33:40.077 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:40.081+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:33:40.078 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:40.081+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:33:40.078 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:40.081+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:33:49.319 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:33:49.320 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:49.327 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:49.336 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:50.314 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:33:50.314 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:33:50.315 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243630 2023-10-14T00:33:50.315 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:50.315 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/60711583 ]] 2023-10-14T00:33:50.315 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/60711583 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\6\0\7\1\1\5\8\3 ]] 2023-10-14T00:33:50.315 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:33:58.935 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:33:58.935 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:33:58.935 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:33:58.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:33:58.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:33:58.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:33:58.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:33:58.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:33:58.937 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:33:58.929+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:34:00.316 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:34:00.317 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:00.317 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:00.318 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:01.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/60711583 2023-10-14T00:34:01.187 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:34:01.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243641 2023-10-14T00:34:01.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:01.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/60711583 ]] 2023-10-14T00:34:01.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/60711583 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\6\0\7\1\1\5\8\3 ]] 2023-10-14T00:34:01.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:34:05.225 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:34:05.225+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:34:11.189 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:34:11.190 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:11.195 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:11.209 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:12.068 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:34:12.073 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:34:12.074 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243652 2023-10-14T00:34:12.075 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:12.075 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/4121420252 ]] 2023-10-14T00:34:12.075 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/4121420252 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\6\0\7\1\1\5\8\3 ]] 2023-10-14T00:34:12.075 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/4121420252 2023-10-14T00:34:14.485 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/4121420252 until 2023-10-14T01:34:13.647915+0000 (3600 sec) 2023-10-14T00:34:14.533 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:34:14.533 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/4121420252 2023-10-14T00:34:15.355 INFO:tasks.workunit.client.0.smithi081.stderr:listed 24 entries 2023-10-14T00:34:15.391 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:34:15.391 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:34:25.402 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:34:25.405 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:25.409 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:25.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:26.422 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:34:26.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:34:26.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243666 2023-10-14T00:34:26.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:26.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/4121420252 ]] 2023-10-14T00:34:26.423 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/4121420252 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\4\1\2\1\4\2\0\2\5\2 ]] 2023-10-14T00:34:26.424 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:34:35.240 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:34:35.240+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:34:35.246 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:34:35.248+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:34:35.246 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:34:35.248+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:34:35.246 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:34:35.248+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:34:36.426 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:34:36.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:36.441 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:36.456 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:37.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:34:37.581 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:34:37.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243677 2023-10-14T00:34:37.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:37.582 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/4121420252 ]] 2023-10-14T00:34:37.582 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/4121420252 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\4\1\2\1\4\2\0\2\5\2 ]] 2023-10-14T00:34:37.582 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:34:47.586 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:34:47.587 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:47.592 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:47.594 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:48.470 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:34:48.471 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:34:48.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243688 2023-10-14T00:34:48.472 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:48.472 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/4121420252 ]] 2023-10-14T00:34:48.472 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/4121420252 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\4\1\2\1\4\2\0\2\5\2 ]] 2023-10-14T00:34:48.472 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:34:58.488 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:58.505 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:34:58.511 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:58.512 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:59.447 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:34:59.447 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:34:59.448 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243699 2023-10-14T00:34:59.448 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:59.448 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/4121420252 ]] 2023-10-14T00:34:59.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/4121420252 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\4\1\2\1\4\2\0\2\5\2 ]] 2023-10-14T00:34:59.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:35:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:34:59.999+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.002 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:34:59.999+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddeb6a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.003+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.003+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf071080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.011+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.011+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4db680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.019+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.019+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf070000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.027+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.027+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.031+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.035+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.039+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.039+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.047+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.047+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde2b5500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.051+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.051+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf6d6a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.059+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.059+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf6d6c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.067+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.067 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.067+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.071+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.071+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf221d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.079+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.079+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.084 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.083+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.084 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.083+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.091+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.091+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.095+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.095+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.099+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.099+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.103 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.103+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.103 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.103+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.103+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.105 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.103+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.107 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.107+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:00.107 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:35:00.107+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:35:05.941 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:05.939+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:35:05.941 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:05.939+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:35:05.942 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:05.939+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:35:05.942 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:05.939+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:35:09.450 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:09.451 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:35:09.451 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:09.471 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:10.401 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/4121420252 2023-10-14T00:35:10.402 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:35:10.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243710 2023-10-14T00:35:10.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:10.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/4121420252 ]] 2023-10-14T00:35:10.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/4121420252 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\4\1\2\1\4\2\0\2\5\2 ]] 2023-10-14T00:35:10.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:35:18.935 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:35:18.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:35:18.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:35:18.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:35:18.936 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:35:18.937 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:35:18.937 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:35:18.937 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:35:18.937 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:35:18.937 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:18.935+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:35:20.405 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:20.405 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:35:20.406 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:20.416 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:21.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:35:21.272 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:35:21.272 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243721 2023-10-14T00:35:21.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:21.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1042564731 ]] 2023-10-14T00:35:21.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1042564731 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\4\1\2\1\4\2\0\2\5\2 ]] 2023-10-14T00:35:21.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/1042564731 2023-10-14T00:35:23.502 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/1042564731 until 2023-10-14T01:35:22.742936+0000 (3600 sec) 2023-10-14T00:35:23.547 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/1042564731 2023-10-14T00:35:23.547 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:35:24.237 INFO:tasks.workunit.client.0.smithi081.stderr:listed 25 entries 2023-10-14T00:35:24.285 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:35:24.285 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:35:34.296 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:35:34.296 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:34.306 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:34.315 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:35.198 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:35:35.198 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:35:35.199 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243735 2023-10-14T00:35:35.199 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:35.200 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1042564731 ]] 2023-10-14T00:35:35.200 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1042564731 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\4\2\5\6\4\7\3\1 ]] 2023-10-14T00:35:35.200 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:35:45.201 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:35:45.202 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:45.205 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:45.205 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:46.130 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:35:46.130 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:35:46.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243746 2023-10-14T00:35:46.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:46.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1042564731 ]] 2023-10-14T00:35:46.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1042564731 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\4\2\5\6\4\7\3\1 ]] 2023-10-14T00:35:46.132 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:35:46.273 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:46.274+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:35:46.273 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:46.274+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:35:46.273 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:46.274+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:35:53.525 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:35:53.526 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:35:53.527 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:35:53.527 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:35:53.527 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:35:53.528 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:35:53.528 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:35:53.528 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:35:53.529 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:35:53.529 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:35:53.529 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:35:53.522+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:35:56.132 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:35:56.133 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:56.134 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:56.138 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:57.182 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:35:57.183 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:35:57.183 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243757 2023-10-14T00:35:57.184 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:57.184 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1042564731 ]] 2023-10-14T00:35:57.185 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1042564731 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\4\2\5\6\4\7\3\1 ]] 2023-10-14T00:35:57.185 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:36:00.000 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.002+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.000 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.002+0000 7f3981387640 -1 librbd::ImageState: 0x563ddeb6a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.014+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.014+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.014+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.014+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf64e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.014+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.014+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.018+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.018+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf64ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.019 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.022+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.022+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf64e880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.026+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.026+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.030+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.029 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.030+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.034+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.035 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.034+0000 7f3981387640 -1 librbd::ImageState: 0x563ddef01000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.044 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.046+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.044 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.046+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.047 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.050+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.050+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf305280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.054+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.054+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf24e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.062+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.062 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.062+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf124a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.070+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.070+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf4daf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.074+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.074+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddef17b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.075 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.078+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.075 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.078+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66dc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.090+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.090+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf221400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.090+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.090+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.098+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.098+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf6d7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.102+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:36:00.102+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf6d6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:07.185 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:07.186 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:07.190 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:36:07.190 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:08.068 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:36:08.069 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:36:08.069 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243768 2023-10-14T00:36:08.069 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:08.070 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1042564731 ]] 2023-10-14T00:36:08.070 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1042564731 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\4\2\5\6\4\7\3\1 ]] 2023-10-14T00:36:08.070 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:36:15.294 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:36:15.297+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:36:18.077 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:18.078 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:18.080 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:18.085 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:36:19.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/1042564731 2023-10-14T00:36:19.044 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:36:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243779 2023-10-14T00:36:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/1042564731 ]] 2023-10-14T00:36:19.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/1042564731 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\4\2\5\6\4\7\3\1 ]] 2023-10-14T00:36:19.046 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:36:25.385 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:36:25.385+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:36:29.056 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:36:29.057 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:29.057 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:29.062 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:30.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:36:30.007 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:36:30.008 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243790 2023-10-14T00:36:30.008 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:30.009 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2894067023 ]] 2023-10-14T00:36:30.009 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2894067023 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\1\0\4\2\5\6\4\7\3\1 ]] 2023-10-14T00:36:30.009 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2894067023 2023-10-14T00:36:32.505 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2894067023 until 2023-10-14T01:36:31.719886+0000 (3600 sec) 2023-10-14T00:36:32.571 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:36:32.577 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2894067023 2023-10-14T00:36:33.382 INFO:tasks.workunit.client.0.smithi081.stderr:listed 26 entries 2023-10-14T00:36:33.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:36:33.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:36:43.421 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:36:43.421 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:43.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:43.444 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:44.416 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:36:44.416 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:36:44.417 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243804 2023-10-14T00:36:44.418 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:44.418 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2894067023 ]] 2023-10-14T00:36:44.418 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2894067023 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\9\4\0\6\7\0\2\3 ]] 2023-10-14T00:36:44.418 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:36:54.420 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:36:54.421 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:54.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:54.443 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:55.306 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:36:55.308 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:36:55.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243815 2023-10-14T00:36:55.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:55.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2894067023 ]] 2023-10-14T00:36:55.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2894067023 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\9\4\0\6\7\0\2\3 ]] 2023-10-14T00:36:55.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:36:59.372 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.372+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:36:59.373 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.372+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:36:59.405 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.404+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:36:59.405 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.404+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:36:59.405 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.404+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:36:59.406 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.404+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:36:59.406 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:36:59.404+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:37:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.008+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.007 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.008+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.008+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf64f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.008+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf304f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.016+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.016+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.016+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.015 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.016+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66cc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.016+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.016+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.036 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.036+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.036+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf322e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.050 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.048+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.050 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.048+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.052+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.052+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d6c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.056+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.056+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.068 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.068+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.068 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.068+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.080+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.080+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.096+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.096+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.117 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.116+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.118 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.116+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.121 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.120+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.121 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.120+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.124 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.124+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.125 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.124+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.130 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.128+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.130 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.128+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.132+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.132+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.136+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.138 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.136+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.149 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.148+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.149 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.148+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.179 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.180+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.179 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:00.180+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.671 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:00.668+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:37:00.672 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:00.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:37:00.672 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:00.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:37:00.672 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:00.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:37:00.673 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:00.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:37:05.311 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:37:05.312 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:05.314 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:05.335 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:06.261 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:37:06.262 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:37:06.262 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243826 2023-10-14T00:37:06.263 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:06.263 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2894067023 ]] 2023-10-14T00:37:06.263 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2894067023 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\9\4\0\6\7\0\2\3 ]] 2023-10-14T00:37:06.263 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:37:16.267 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:16.270 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:16.286 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:16.287 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:37:17.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:37:17.194 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:37:17.197 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243837 2023-10-14T00:37:17.197 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:17.197 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2894067023 ]] 2023-10-14T00:37:17.197 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2894067023 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\9\4\0\6\7\0\2\3 ]] 2023-10-14T00:37:17.197 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:37:19.097 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:37:19.092+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:37:27.203 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:37:27.204 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:27.214 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:27.215 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:28.109 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2894067023 2023-10-14T00:37:28.109 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:37:28.109 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243848 2023-10-14T00:37:28.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:28.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2894067023 ]] 2023-10-14T00:37:28.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2894067023 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\9\4\0\6\7\0\2\3 ]] 2023-10-14T00:37:28.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:37:38.113 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:38.114 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:37:38.114 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:38.121 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:39.025 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:37:39.028 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:37:39.029 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243859 2023-10-14T00:37:39.029 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:39.029 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3155963726 ]] 2023-10-14T00:37:39.030 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3155963726 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\8\9\4\0\6\7\0\2\3 ]] 2023-10-14T00:37:39.030 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3155963726 2023-10-14T00:37:41.066 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3155963726 until 2023-10-14T01:37:40.303742+0000 (3600 sec) 2023-10-14T00:37:41.110 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:37:41.111 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3155963726 2023-10-14T00:37:41.911 INFO:tasks.workunit.client.0.smithi081.stderr:listed 27 entries 2023-10-14T00:37:41.944 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:37:41.944 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:37:51.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:51.947 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:37:51.951 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:51.962 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:52.470 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:52.467+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:37:52.474 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:52.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:37:52.474 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:52.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:37:52.474 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:37:52.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:37:52.901 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:37:52.901 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:37:52.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243872 2023-10-14T00:37:52.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:52.903 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3155963726 ]] 2023-10-14T00:37:52.903 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3155963726 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\5\5\9\6\3\7\2\6 ]] 2023-10-14T00:37:52.903 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:38:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:59.999+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.001 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:37:59.999+0000 7f3981387640 -1 librbd::ImageState: 0x563ddef01e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf66d980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf292300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.007+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.007+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf66c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.012 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.011+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.013 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.011+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf64ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.015+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.015+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.019+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.019+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.023+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.026 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.023+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.039+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.039+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.043+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.043+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.051+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.051+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.051+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.051+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.067+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.067+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86fc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.071 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.071+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.071+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.076 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.075+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.075+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7ba000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.079+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.081 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.079+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7ba200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.083+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.086 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.083+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7ba400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.091+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.091+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7ba600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.099+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.100 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.099+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7ba800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.103+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:38:00.103+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7baa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:02.904 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:38:02.909 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:02.920 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:02.923 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:03.761 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:38:03.762 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:38:03.762 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243883 2023-10-14T00:38:03.762 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:03.763 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3155963726 ]] 2023-10-14T00:38:03.763 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3155963726 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\5\5\9\6\3\7\2\6 ]] 2023-10-14T00:38:03.763 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:38:04.786 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.783+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:38:04.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.795+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:38:04.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.795+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:38:04.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.795+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:38:04.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.795+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:38:04.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.795+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:38:04.813 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:04.811+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:38:07.413 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.411+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:38:07.581 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.579+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:38:07.582 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.579+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:38:07.589 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.583+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:38:07.589 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.583+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:38:07.589 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.583+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:38:07.590 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.583+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:38:07.590 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:07.587+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:38:13.767 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:13.769 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:38:13.782 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:13.788 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:14.671 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:38:14.674 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:38:14.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243894 2023-10-14T00:38:14.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:14.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3155963726 ]] 2023-10-14T00:38:14.676 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3155963726 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\5\5\9\6\3\7\2\6 ]] 2023-10-14T00:38:14.676 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:38:24.678 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:38:24.678 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:24.679 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:24.679 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:24.961 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:24.950+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:38:24.967 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:24.958+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:38:24.967 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:24.958+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:38:24.967 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:24.958+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:38:25.598 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:38:25.599 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:38:25.600 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243905 2023-10-14T00:38:25.600 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:25.600 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3155963726 ]] 2023-10-14T00:38:25.600 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3155963726 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\5\5\9\6\3\7\2\6 ]] 2023-10-14T00:38:25.601 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:38:35.602 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:38:35.602 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:35.606 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:35.612 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:36.528 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3155963726 2023-10-14T00:38:36.528 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:38:36.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243916 2023-10-14T00:38:36.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:36.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3155963726 ]] 2023-10-14T00:38:36.529 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3155963726 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\5\5\9\6\3\7\2\6 ]] 2023-10-14T00:38:36.530 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:38:46.534 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:46.535 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:46.535 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:46.537 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:38:47.516 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:38:47.519 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:38:47.520 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243927 2023-10-14T00:38:47.520 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:47.520 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3240612452 ]] 2023-10-14T00:38:47.521 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3240612452 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\1\5\5\9\6\3\7\2\6 ]] 2023-10-14T00:38:47.521 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3240612452 2023-10-14T00:38:50.053 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3240612452 until 2023-10-14T01:38:49.279924+0000 (3600 sec) 2023-10-14T00:38:50.093 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:38:50.094 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3240612452 2023-10-14T00:38:50.827 INFO:tasks.workunit.client.0.smithi081.stderr:listed 28 entries 2023-10-14T00:38:50.867 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:38:50.868 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:38:58.469 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:38:58.470 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:38:58.470 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:38:58.470 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:38:58.470 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:38:58.471 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:38:58.471 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:38:58.471 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:38:58.471 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:38:58.469+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:39:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf054a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf322200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.009+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.013+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.013+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf6d6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.017+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.018 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.017+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.021+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.023 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.021+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bb200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.025+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.029+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bb080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.033+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.033+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.036 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.037+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.036 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.037+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bb600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.040 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.041+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.041 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.041+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.045 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.045+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.045+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.053+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.053+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.057+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.057+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bbc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.061+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.061 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.061+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bbe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.065+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.065+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.087 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.085+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.087 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.085+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.090 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.089+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.089+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.097+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.097+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.101+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:39:00.101+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.869 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:39:00.870 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:00.871 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:00.872 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:01.722 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:39:01.725 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:39:01.727 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243941 2023-10-14T00:39:01.727 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:01.727 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3240612452 ]] 2023-10-14T00:39:01.728 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3240612452 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\0\6\1\2\4\5\2 ]] 2023-10-14T00:39:01.728 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:39:11.728 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:39:11.728 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:11.729 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:11.729 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:12.649 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:39:12.649 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:39:12.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243952 2023-10-14T00:39:12.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:12.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3240612452 ]] 2023-10-14T00:39:12.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3240612452 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\0\6\1\2\4\5\2 ]] 2023-10-14T00:39:12.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:39:22.652 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:22.655 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:22.660 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:39:22.669 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:23.536 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:39:23.536 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:39:23.537 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243963 2023-10-14T00:39:23.537 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:23.537 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3240612452 ]] 2023-10-14T00:39:23.538 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3240612452 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\0\6\1\2\4\5\2 ]] 2023-10-14T00:39:23.538 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:39:33.540 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:33.541 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:39:33.541 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:33.544 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:34.435 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:39:34.436 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:39:34.436 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243974 2023-10-14T00:39:34.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:34.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3240612452 ]] 2023-10-14T00:39:34.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3240612452 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\0\6\1\2\4\5\2 ]] 2023-10-14T00:39:34.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:39:44.438 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:44.439 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:44.448 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:39:44.452 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:45.364 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3240612452 2023-10-14T00:39:45.364 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:39:45.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243985 2023-10-14T00:39:45.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:45.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3240612452 ]] 2023-10-14T00:39:45.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3240612452 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\0\6\1\2\4\5\2 ]] 2023-10-14T00:39:45.366 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:39:55.368 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:55.369 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:55.370 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:39:55.389 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:56.282 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:39:56.282 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:39:56.283 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697243996 2023-10-14T00:39:56.283 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:56.283 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/880919163 ]] 2023-10-14T00:39:56.284 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/880919163 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\2\4\0\6\1\2\4\5\2 ]] 2023-10-14T00:39:56.284 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/880919163 2023-10-14T00:39:58.086 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/880919163 until 2023-10-14T01:39:57.398340+0000 (3600 sec) 2023-10-14T00:39:58.121 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:39:58.122 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/880919163 2023-10-14T00:39:58.896 INFO:tasks.workunit.client.0.smithi081.stderr:listed 29 entries 2023-10-14T00:39:58.934 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:39:58.934 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:40:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf86f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.003+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.005 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.003+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf66c200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.015+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.019+0000 7f3981387640 -1 librbd::ImageState: 0x563ddeb6a400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.019+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.019+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf66dc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.023+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.023+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf24e180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.029 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.027+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.029 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.027+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf6d6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.055+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.055+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf304f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.063+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.065 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.063+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b5580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.068 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.067+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.068 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.067+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7bb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.071+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.072 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.071+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7ba200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.075+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.075+0000 7f3981387640 -1 librbd::ImageState: 0x563dde2b4080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.091+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.094 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.091+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf4db380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.095+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.098 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.095+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7bad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.099+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.102 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.099+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7bbe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.105 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.103+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.103+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7bbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.108 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.107+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.108 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.107+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf816580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.111 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.107+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.111 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.107+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf816200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.114 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.111+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.111+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf7bac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.118 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.115+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.119 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.115+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf816b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.123 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.119+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:00.124 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:40:00.119+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf816980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:40:01.850 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:40:01.851 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.843+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:40:01.851 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.847+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:40:01.852 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.847+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:40:01.852 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.847+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:40:01.858 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.855+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:40:01.880 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.867+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:40:01.880 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.867+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:40:01.881 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.867+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:40:01.888 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.883+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:40:01.888 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.883+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:40:01.888 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.883+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:40:01.888 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.883+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:40:01.888 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.883+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:40:01.890 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.887+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:40:01.890 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:40:01.887+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:40:08.938 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:08.941 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:08.941 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:40:08.942 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:09.887 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:40:09.887 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:40:09.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244009 2023-10-14T00:40:09.888 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:09.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/880919163 ]] 2023-10-14T00:40:09.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/880919163 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\8\0\9\1\9\1\6\3 ]] 2023-10-14T00:40:09.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:40:19.895 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:40:19.895 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:19.903 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:19.907 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:20.739 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:40:20.739 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:40:20.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244020 2023-10-14T00:40:20.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:20.740 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/880919163 ]] 2023-10-14T00:40:20.741 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/880919163 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\8\0\9\1\9\1\6\3 ]] 2023-10-14T00:40:20.741 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:40:30.742 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:30.742 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:30.744 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:30.758 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:40:31.673 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:40:31.673 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:40:31.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244031 2023-10-14T00:40:31.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:31.674 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/880919163 ]] 2023-10-14T00:40:31.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/880919163 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\8\0\9\1\9\1\6\3 ]] 2023-10-14T00:40:31.675 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:40:41.676 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:41.677 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:41.680 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:41.686 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:40:42.540 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:40:42.540 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:40:42.541 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244042 2023-10-14T00:40:42.541 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:42.541 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/880919163 ]] 2023-10-14T00:40:42.541 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/880919163 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\8\0\9\1\9\1\6\3 ]] 2023-10-14T00:40:42.542 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:40:52.544 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:40:52.551 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:52.560 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:52.561 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:53.445 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/880919163 2023-10-14T00:40:53.447 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:40:53.448 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244053 2023-10-14T00:40:53.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:53.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/880919163 ]] 2023-10-14T00:40:53.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/880919163 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\8\0\9\1\9\1\6\3 ]] 2023-10-14T00:40:53.449 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:41:03.456 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:03.463 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:03.464 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:03.464 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:41:04.392 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:41:04.392 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:41:04.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244064 2023-10-14T00:41:04.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:04.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:41:04.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\8\8\0\9\1\9\1\6\3 ]] 2023-10-14T00:41:04.394 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/3013418471 2023-10-14T00:41:06.768 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/3013418471 until 2023-10-14T01:41:06.020830+0000 (3600 sec) 2023-10-14T00:41:06.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:41:06.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/3013418471 2023-10-14T00:41:07.552 INFO:tasks.workunit.client.0.smithi081.stderr:listed 30 entries 2023-10-14T00:41:07.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:41:07.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:41:17.611 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:41:17.611 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:17.611 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:17.612 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:18.506 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:41:18.507 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:41:18.507 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244078 2023-10-14T00:41:18.507 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:18.507 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:41:18.508 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:41:18.508 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:41:28.513 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:41:28.513 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:28.514 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:28.532 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:29.568 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:41:29.568 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:41:29.569 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244089 2023-10-14T00:41:29.569 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:29.569 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:41:29.570 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:41:29.570 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:41:39.573 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:39.578 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:41:39.584 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:39.584 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:40.535 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:41:40.535 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:41:40.536 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244100 2023-10-14T00:41:40.536 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:40.536 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:41:40.536 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:41:40.536 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:41:50.152 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:41:50.149+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:41:50.541 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:50.541 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:50.541 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:50.549 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:41:51.467 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:41:51.467 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:41:51.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244111 2023-10-14T00:41:51.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:51.469 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:41:51.469 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:41:51.469 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:41:52.399 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:41:52.397+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:42:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.016 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf305100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf86ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf64e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.017 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.013+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf64ec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.017+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.017+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf322500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.021+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.021 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.021+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7ba000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.025+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.024 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.025+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.033+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.034 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.033+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf66d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.037+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.038 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.037+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.041+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.042 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.041+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.045+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.046 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.045+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.049+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.051 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.049+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7bbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.053+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.053+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf054700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.057+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.057 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.057+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf64f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.057+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.060 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.057+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf323680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.065 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.061+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.061+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.065+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.066 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.065+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf816d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.069+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.069+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf7ba000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.073+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.074 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.073+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf817380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.077+0000 7f3980385640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:00.077+0000 7f3980385640 -1 librbd::ImageState: 0x563ddf817200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:01.471 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:01.473 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:01.476 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:01.494 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:42:02.380 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:42:02.380 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:42:02.380 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244122 2023-10-14T00:42:02.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:02.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:42:02.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:42:02.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:42:09.649 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:42:09.649 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:42:09.649 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:42:09.650 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:42:09.650 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:42:09.650 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:42:09.650 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:42:09.650 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:42:09.651 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:42:09.651 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:09.644+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:42:12.015 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.012+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:42:12.015 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.012+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:42:12.023 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.016+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:42:12.025 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:42:12.025 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:42:12.025 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:42:12.025 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:42:12.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:42:12.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:42:12.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:42:12.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.024+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:42:12.033 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:42:12.028+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:42:12.382 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:42:12.384 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:12.384 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:12.396 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:13.409 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/3013418471 2023-10-14T00:42:13.410 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:42:13.410 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244133 2023-10-14T00:42:13.410 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:13.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/3013418471 ]] 2023-10-14T00:42:13.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/3013418471 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:42:13.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:42:23.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:42:23.423 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:23.423 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:23.423 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:24.433 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:42:24.434 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:42:24.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244144 2023-10-14T00:42:24.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:24.434 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2292754641 ]] 2023-10-14T00:42:24.435 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2292754641 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\3\0\1\3\4\1\8\4\7\1 ]] 2023-10-14T00:42:24.435 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2292754641 2023-10-14T00:42:26.105 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2292754641 until 2023-10-14T01:42:25.347565+0000 (3600 sec) 2023-10-14T00:42:26.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:42:26.132 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2292754641 2023-10-14T00:42:26.794 INFO:tasks.workunit.client.0.smithi081.stderr:listed 31 entries 2023-10-14T00:42:26.822 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:42:26.823 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:42:36.823 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:36.824 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:36.824 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:42:36.837 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:36.879 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:42:36.876+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:42:37.633 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:42:37.633 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:42:37.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244157 2023-10-14T00:42:37.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:37.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2292754641 ]] 2023-10-14T00:42:37.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2292754641 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\9\2\7\5\4\6\4\1 ]] 2023-10-14T00:42:37.635 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:42:39.300 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:42:39.296+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:42:47.636 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:42:47.636 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:47.636 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:47.637 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:48.565 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:42:48.566 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:42:48.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244168 2023-10-14T00:42:48.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:48.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2292754641 ]] 2023-10-14T00:42:48.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2292754641 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\9\2\7\5\4\6\4\1 ]] 2023-10-14T00:42:48.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:42:50.520 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:42:50.516+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:42:58.581 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:42:58.582 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:58.596 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:58.597 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:59.517 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:42:59.517 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:42:59.518 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244179 2023-10-14T00:42:59.519 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:59.519 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2292754641 ]] 2023-10-14T00:42:59.519 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2292754641 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\9\2\7\5\4\6\4\1 ]] 2023-10-14T00:42:59.519 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:43:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:59.999+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.003 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:42:59.999+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf6d7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.003+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.003+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.007+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.007+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddea1f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.023+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.023+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf66d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.039+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.043 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.039+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf64f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.054 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.051+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.055 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.051+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf323b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.067+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.067+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf7bb600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.075+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.079 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.075+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf293c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.084 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.079+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.084 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.079+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.095+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.095+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.105 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.103+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.105 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.103+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf86fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.109 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.107+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.110 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.107+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf7bbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.111+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.111+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.120 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.115+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.121 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.115+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.124 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.119+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.124 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.119+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.128 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.123+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.129 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.123+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.131+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.133 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.131+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.135+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.135+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.141 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.139+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.141 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.139+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.145 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.143+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.145 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:43:00.143+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:09.534 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:43:09.544 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:09.544 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:09.547 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:10.542 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:43:10.543 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:43:10.543 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244190 2023-10-14T00:43:10.543 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:10.544 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2292754641 ]] 2023-10-14T00:43:10.544 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2292754641 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\9\2\7\5\4\6\4\1 ]] 2023-10-14T00:43:10.544 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:43:17.348 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.343+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:43:17.354 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.351+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:43:17.355 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.351+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:43:17.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:43:17.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:43:17.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:43:17.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:43:17.380 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:43:17.380 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:17.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:43:20.545 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:20.546 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:20.546 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:20.546 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:43:21.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2292754641 2023-10-14T00:43:21.502 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:43:21.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244201 2023-10-14T00:43:21.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:21.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2292754641 ]] 2023-10-14T00:43:21.503 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2292754641 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\9\2\7\5\4\6\4\1 ]] 2023-10-14T00:43:21.504 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:43:31.514 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:43:31.515 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:31.524 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:31.525 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:32.335 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:43:32.336 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:43:32.336 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244212 2023-10-14T00:43:32.337 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:32.337 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2286238291 ]] 2023-10-14T00:43:32.337 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2286238291 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\9\2\7\5\4\6\4\1 ]] 2023-10-14T00:43:32.337 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/2286238291 2023-10-14T00:43:34.670 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/2286238291 until 2023-10-14T01:43:33.880114+0000 (3600 sec) 2023-10-14T00:43:34.721 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:43:34.721 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/2286238291 2023-10-14T00:43:35.452 INFO:tasks.workunit.client.0.smithi081.stderr:listed 32 entries 2023-10-14T00:43:35.482 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:43:35.482 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:43:45.304 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.294+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:43:45.304 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.294+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:43:45.304 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.294+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:43:45.307 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:43:45.308 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:43:45.308 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.302+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:43:45.325 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:45.322+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:43:45.489 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:43:45.490 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:45.490 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:45.511 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:46.372 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:43:46.373 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:43:46.380 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244226 2023-10-14T00:43:46.380 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:46.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2286238291 ]] 2023-10-14T00:43:46.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2286238291 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\8\6\2\3\8\2\9\1 ]] 2023-10-14T00:43:46.381 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:43:56.385 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:56.389 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:43:56.403 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:56.403 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:56.517 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:43:56.510+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:43:57.279 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:43:57.279 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:43:57.280 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244237 2023-10-14T00:43:57.280 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:57.280 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2286238291 ]] 2023-10-14T00:43:57.280 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2286238291 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\8\6\2\3\8\2\9\1 ]] 2023-10-14T00:43:57.281 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:44:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.006+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.006+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.006+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf220f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.010 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.006+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf305280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.006+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.011 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.006+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.010+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.014 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.010+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf304f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.019 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.014+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.020 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.014+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf86f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.034+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.039 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.034+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf86ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.042+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.042+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.063 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.058+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.064 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.058+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.066+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.070 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.066+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.080 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.074+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.080 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.074+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.096 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.090+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.097 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.090+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48a900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.099 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.094+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.100 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.094+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.107 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.102+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.108 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.102+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48ad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.115 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.110+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.110+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.110+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.116 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.110+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.130+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.137 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.130+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.139 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.134+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.139 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.134+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.140 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.138+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.141 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.138+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48b700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.144 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.138+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.144 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.138+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.148 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.142+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:00.149 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:44:00.142+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:44:07.282 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:07.292 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:44:07.294 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:07.295 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:08.223 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:44:08.224 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:44:08.224 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244248 2023-10-14T00:44:08.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:08.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2286238291 ]] 2023-10-14T00:44:08.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2286238291 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\8\6\2\3\8\2\9\1 ]] 2023-10-14T00:44:08.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:44:18.228 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:44:18.228 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:18.236 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:18.237 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:19.150 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:44:19.152 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:44:19.153 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244259 2023-10-14T00:44:19.153 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:19.154 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2286238291 ]] 2023-10-14T00:44:19.154 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2286238291 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\8\6\2\3\8\2\9\1 ]] 2023-10-14T00:44:19.154 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:44:29.161 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:29.161 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:29.162 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:44:29.164 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:30.054 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/2286238291 2023-10-14T00:44:30.054 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:44:30.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244270 2023-10-14T00:44:30.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:30.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/2286238291 ]] 2023-10-14T00:44:30.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/2286238291 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\8\6\2\3\8\2\9\1 ]] 2023-10-14T00:44:30.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:44:40.074 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:44:40.074 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:40.075 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:40.077 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:40.972 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:44:40.972 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:44:40.973 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244280 2023-10-14T00:44:40.973 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:40.973 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:44:40.974 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\2\2\8\6\2\3\8\2\9\1 ]] 2023-10-14T00:44:40.974 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist add 172.21.15.81:0/539853599 2023-10-14T00:44:42.674 INFO:tasks.workunit.client.0.smithi081.stderr:blocklisting 172.21.15.81:0/539853599 until 2023-10-14T01:44:41.910057+0000 (3600 sec) 2023-10-14T00:44:42.709 INFO:tasks.workunit.client.0.smithi081.stderr:+ ceph osd blocklist ls 2023-10-14T00:44:42.709 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep -q 172.21.15.81:0/539853599 2023-10-14T00:44:43.470 INFO:tasks.workunit.client.0.smithi081.stderr:listed 33 entries 2023-10-14T00:44:43.504 INFO:tasks.workunit.client.0.smithi081.stderr:+ PREV_CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:44:43.504 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:44:48.345 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:44:48.345+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:44:52.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:44:52.797+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:44:52.806 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:44:52.805+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:44:52.809 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:44:52.809+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:44:53.510 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:44:53.511 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:53.519 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:53.534 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:54.388 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:44:54.389 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:44:54.389 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244294 2023-10-14T00:44:54.390 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:54.390 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:44:54.390 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:44:54.390 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:45:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.001+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.004 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.001+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf66c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.005+0000 7f3981387640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.005+0000 7f3981387640 -1 librbd::ImageState: 0x563ddf323e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.009+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.009+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf817f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.021+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.022 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.021+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf816c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.028 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.025+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.029 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.025+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.033+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.033 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.033+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48bb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.037+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.037 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.037+0000 7f3980b86640 -1 librbd::ImageState: 0x563dde48bc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.048 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.045+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.049 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.045+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.053+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.053 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.053+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.069+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.069 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.069+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.073+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.073 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.073+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.077+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.077 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.077+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.085+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.088 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.085+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.089+0000 7f3980b86640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:00.091 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T00:45:00.089+0000 7f3980b86640 -1 librbd::ImageState: 0x563ddf260c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:45:04.410 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:45:04.412 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:04.412 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:04.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:05.272 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:45:05.272 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:45:05.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244305 2023-10-14T00:45:05.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:05.274 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:45:05.274 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:45:05.274 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:45:07.414 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:45:07.415 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:45:07.423 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:45:07.423 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:45:07.424 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:45:07.424 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:45:07.424 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:45:07.424 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:45:07.424 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:07.421+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:45:15.275 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:15.282 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:45:15.295 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:15.303 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:16.167 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:45:16.167 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:45:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244316 2023-10-14T00:45:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:45:16.168 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:45:16.169 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:45:26.170 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:26.171 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:45:26.182 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:26.187 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:27.034 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:45:27.034 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:45:27.035 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244327 2023-10-14T00:45:27.035 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:27.035 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:45:27.036 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:45:27.036 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:45:28.442 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:45:28.443 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:45:28.443 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:45:28.443 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T00:45:28.443 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:45:28.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:45:28.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:45:28.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:45:28.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:45:28.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:45:28.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:45:28.445 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:45:28.445 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:45:28.445 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:45:28.445 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:45:28.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:45:37.037 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:37.037 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:37.042 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:37.044 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:45:37.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:45:37.945 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:45:37.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244337 2023-10-14T00:45:37.946 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:37.946 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:45:37.946 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:45:37.946 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:45:47.940 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:45:47.941 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:47.941 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:47.941 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:48.912 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:45:48.913 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:45:48.913 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244348 2023-10-14T00:45:48.914 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:48.914 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:45:48.914 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:45:48.914 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:45:58.917 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:45:58.924 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:58.924 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:58.944 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:59.863 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:45:59.863 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:45:59.864 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244359 2023-10-14T00:45:59.864 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:59.865 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:45:59.865 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:45:59.865 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:46:07.866 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:07.860+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:46:07.876 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:07.872+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:46:07.876 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:07.872+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:46:07.876 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:07.872+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:46:09.883 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:09.883 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:09.886 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:09.889 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:46:10.821 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:46:10.821 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:46:10.822 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244370 2023-10-14T00:46:10.823 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:10.823 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:46:10.823 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:46:10.823 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:46:12.675 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:46:12.683 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:46:12.683 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.672+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:46:12.719 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:46:12.719 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:46:12.719 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:46:12.720 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:46:12.720 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:46:12.720 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:46:12.720 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:46:12.720 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:12.716+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:46:20.824 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:46:20.825 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:20.825 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:20.829 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:21.811 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:46:21.811 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:46:21.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244381 2023-10-14T00:46:21.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:21.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:46:21.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:46:21.813 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:46:31.814 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:31.816 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:46:31.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:31.829 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:32.817 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:46:32.817 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:46:32.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244392 2023-10-14T00:46:32.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:32.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:46:32.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:46:32.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:46:39.350 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:46:39.347+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:46:42.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:46:42.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:42.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:42.820 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:43.687 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:46:43.687 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:46:43.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244403 2023-10-14T00:46:43.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:43.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:46:43.689 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:46:43.689 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:46:53.689 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:46:53.690 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:53.695 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:53.707 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:54.633 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:46:54.694 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:46:54.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244414 2023-10-14T00:46:54.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:54.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:46:54.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:46:54.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:46:58.976 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.971+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:46:58.977 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.971+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:46:58.977 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.971+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:46:58.989 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.987+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:46:58.990 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.987+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:46:58.990 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.987+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:46:58.990 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.987+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:46:58.992 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:46:58.987+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:47:04.642 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:04.643 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:04.657 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:04.659 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:47:05.626 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:47:05.627 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:47:05.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244425 2023-10-14T00:47:05.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:05.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:47:05.634 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:47:05.635 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:47:15.643 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:47:15.643 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:15.644 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:15.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:16.679 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:47:16.680 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:47:16.681 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244436 2023-10-14T00:47:16.681 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:16.681 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:47:16.681 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:47:16.681 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:47:26.683 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:26.685 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:47:26.686 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:26.693 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:27.595 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:47:27.595 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:47:27.596 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244447 2023-10-14T00:47:27.596 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:27.596 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:47:27.596 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:47:27.597 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:47:31.974 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:31.974+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:47:31.974 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:31.974+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:47:31.974 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:31.974+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:47:31.975 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:31.974+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:47:31.975 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:31.974+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:47:37.597 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:47:37.598 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:37.617 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:37.620 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:38.475 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:47:38.475 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:47:38.475 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244458 2023-10-14T00:47:38.476 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:38.476 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:47:38.476 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:47:38.476 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:47:48.477 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:47:48.478 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:48.481 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:48.482 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:49.421 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:47:49.423 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:47:49.424 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244469 2023-10-14T00:47:49.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:49.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:47:49.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:47:49.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:47:54.063 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.062+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:47:54.072 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.066+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:47:54.073 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.066+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:47:54.073 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.066+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:47:54.073 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.066+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:47:54.074 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.074+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:47:54.074 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.074+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:47:54.074 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.074+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:47:54.467 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.465+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:47:54.467 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.465+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:47:54.467 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.465+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:47:54.467 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.465+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:47:54.469 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.465+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:47:54.478 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:47:54.477+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:47:59.426 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:47:59.429 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:59.435 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:59.438 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:00.353 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:48:00.353 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:48:00.354 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244480 2023-10-14T00:48:00.354 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:00.355 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:48:00.355 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:48:00.355 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:48:08.554 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:48:08.545+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:48:08.555 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:48:08.545+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:48:10.361 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:48:10.361 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:10.362 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:10.363 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:11.231 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:48:11.231 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:48:11.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244491 2023-10-14T00:48:11.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:11.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:48:11.232 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:48:11.233 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:48:21.235 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:21.236 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:21.242 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:48:21.250 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:22.234 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:48:22.234 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:48:22.236 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244502 2023-10-14T00:48:22.236 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:22.237 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:48:22.237 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:48:22.237 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:48:31.681 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:48:31.681+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:48:32.239 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:48:32.241 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:32.243 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:32.253 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:33.167 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:48:33.171 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:48:33.173 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244513 2023-10-14T00:48:33.173 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:33.174 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:48:33.174 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:48:33.174 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:48:43.175 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:48:43.175 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:43.176 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:43.184 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:43.941 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:48:43.944 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:48:43.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244523 2023-10-14T00:48:43.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:43.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:48:43.945 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:48:43.946 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:48:53.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:53.948 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:53.948 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:48:53.954 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:54.781 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:48:54.781 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:48:54.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244534 2023-10-14T00:48:54.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:54.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:48:54.783 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:48:54.783 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:49:04.785 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:04.786 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:04.793 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:49:04.797 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:05.770 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:49:05.770 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:49:05.771 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244545 2023-10-14T00:49:05.771 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:05.771 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:49:05.772 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:49:05.772 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:49:15.777 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:49:15.778 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:15.782 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:15.802 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:16.666 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:49:16.666 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:49:16.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244556 2023-10-14T00:49:16.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:16.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:49:16.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:49:16.668 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:49:26.676 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:26.676 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:26.676 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:49:26.693 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:27.687 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:49:27.687 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:49:27.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244567 2023-10-14T00:49:27.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:27.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:49:27.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:49:27.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:49:37.691 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:49:37.691 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:37.705 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:37.716 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:38.714 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:49:38.715 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:49:38.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244578 2023-10-14T00:49:38.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:38.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:49:38.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:49:38.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:49:41.371 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.355+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:49:41.372 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.355+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:49:41.372 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.355+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:49:41.376 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:49:41.377 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:49:41.377 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:49:41.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:49:41.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.375+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:49:41.384 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:49:41.379+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T00:49:48.719 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:49:48.719 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:48.729 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:48.731 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:49.570 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:49:49.571 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:49:49.572 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244589 2023-10-14T00:49:49.572 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:49.572 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:49:49.572 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:49:49.572 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:49:59.573 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:49:59.573 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:59.574 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:59.586 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:00.566 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:50:00.566 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:50:00.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244600 2023-10-14T00:50:00.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:00.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:50:00.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:50:00.568 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:50:08.481 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:50:08.475+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:50:10.569 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:10.569 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:10.572 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:50:10.577 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:11.482 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:50:11.482 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:50:11.483 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244611 2023-10-14T00:50:11.483 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:11.483 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:50:11.483 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:50:11.484 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:50:17.520 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:50:17.514+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:50:21.485 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:21.486 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:50:21.495 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:21.495 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:22.467 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:50:22.467 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:50:22.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244622 2023-10-14T00:50:22.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:22.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:50:22.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:50:22.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:50:32.471 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:32.472 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:32.473 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:32.480 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:50:33.424 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:50:33.424 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:50:33.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244633 2023-10-14T00:50:33.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:33.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:50:33.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:50:33.425 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:50:43.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:50:43.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:43.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:43.441 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:44.007 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:50:44.006+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:50:44.300 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:50:44.300 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:50:44.301 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244644 2023-10-14T00:50:44.301 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:44.301 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:50:44.301 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:50:44.302 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:50:54.306 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:54.306 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:54.326 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:54.330 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:50:55.422 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:50:55.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:50:55.426 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244655 2023-10-14T00:50:55.426 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:55.427 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:50:55.427 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:50:55.427 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:51:05.430 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:51:05.441 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:05.441 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:05.452 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:06.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:51:06.311 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:51:06.312 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244666 2023-10-14T00:51:06.312 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:06.312 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:51:06.313 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:51:06.313 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:51:16.314 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:51:16.315 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:16.315 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:16.319 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:17.230 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:51:17.234 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:51:17.235 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244677 2023-10-14T00:51:17.236 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:17.236 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:51:17.236 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:51:17.236 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:51:27.247 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:27.250 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:27.255 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:27.263 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:51:28.157 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:51:28.159 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:51:28.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244688 2023-10-14T00:51:28.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:28.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:51:28.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:51:28.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:51:38.169 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:51:38.177 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:38.178 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:38.184 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:39.083 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:51:39.083 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:51:39.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244699 2023-10-14T00:51:39.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:39.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:51:39.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:51:39.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:51:43.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:51:43.241+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:51:43.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:51:43.241+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:51:43.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:51:43.245+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:51:43.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:51:43.245+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:51:43.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:51:43.245+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:51:43.272 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:51:43.269+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:51:49.085 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:51:49.086 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:49.104 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:49.104 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:50.067 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:51:50.069 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:51:50.070 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244710 2023-10-14T00:51:50.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:50.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:51:50.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:51:50.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:52:00.084 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:52:00.085 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:00.085 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:00.097 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:00.601 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:00.596+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:52:00.607 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:00.604+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:52:00.607 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:00.604+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:52:01.039 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:52:01.039 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:52:01.040 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244721 2023-10-14T00:52:01.040 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:01.040 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:52:01.040 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:52:01.041 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:52:11.041 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:52:11.042 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:11.043 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:11.045 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:12.042 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:52:12.042 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:52:12.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244732 2023-10-14T00:52:12.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:12.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:52:12.044 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:52:12.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:52:22.049 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:52:22.055 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:22.065 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:22.065 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:22.956 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:52:22.957 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:52:22.957 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244742 2023-10-14T00:52:22.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:22.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:52:22.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:52:22.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:52:32.328 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.324+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:52:32.328 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.324+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:52:32.336 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:52:32.337 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:52:32.337 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:52:32.337 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:52:32.337 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:52:32.337 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:52:32.338 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:32.332+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:52:32.968 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:32.969 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:32.983 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:32.989 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:52:33.957 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:52:33.957 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:52:33.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244753 2023-10-14T00:52:33.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:33.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:52:33.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:52:33.958 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:52:43.960 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:52:43.960 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:43.960 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:43.974 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:44.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:52:44.889 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:52:44.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244764 2023-10-14T00:52:44.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:44.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:52:44.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:52:44.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:52:53.205 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:52:53.206 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:52:53.206 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:52:53.206 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:52:53.206 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:52:53.207 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:52:53.207 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:52:53.207 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:52:53.207 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:52:53.200+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:52:54.899 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:52:54.900 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:54.900 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:54.909 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:55.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:52:55.903 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:52:55.903 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244775 2023-10-14T00:52:55.903 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:55.904 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:52:55.904 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:52:55.904 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:53:05.907 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:05.908 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:53:05.909 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:05.917 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:06.889 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:53:06.890 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:53:06.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244786 2023-10-14T00:53:06.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:06.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:53:06.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:53:06.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:53:09.295 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:53:09.296 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:53:09.296 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:53:09.296 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:53:09.296 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T00:53:09.297 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:53:09.297 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:53:09.297 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:53:09.297 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:53:09.297 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:53:09.298 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:09.291+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:53:16.899 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:53:16.900 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:16.903 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:16.909 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:17.917 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:53:17.917 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:53:17.918 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244797 2023-10-14T00:53:17.918 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:17.918 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:53:17.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:53:17.919 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:53:27.922 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:27.930 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:27.933 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:53:27.947 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:28.921 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:53:28.922 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:53:28.922 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244808 2023-10-14T00:53:28.923 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:28.923 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:53:28.923 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:53:28.923 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:53:38.227 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.223+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:53:38.227 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.223+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:53:38.242 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.235+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:53:38.242 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.235+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:53:38.242 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.235+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:53:38.243 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.235+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:53:38.243 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:53:38.235+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:53:38.925 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:53:38.926 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:38.927 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:38.929 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:39.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:53:39.890 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:53:39.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244819 2023-10-14T00:53:39.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:39.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:53:39.891 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:53:39.892 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:53:49.894 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:49.897 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:53:49.906 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:49.911 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:50.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:53:50.809 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:53:50.810 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244830 2023-10-14T00:53:50.810 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:50.810 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:53:50.810 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:53:50.811 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:54:00.812 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:00.812 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:54:00.813 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:00.815 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:01.741 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:54:01.741 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:54:01.742 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244841 2023-10-14T00:54:01.742 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:01.742 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:54:01.743 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:54:01.743 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:54:01.796 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.790+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:54:01.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:54:01.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:54:01.801 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:54:01.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:54:01.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:54:01.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:54:01.802 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:54:01.794+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:54:11.759 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:11.759 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:11.768 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:11.780 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:54:12.746 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:54:12.746 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:54:12.747 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244852 2023-10-14T00:54:12.747 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:12.748 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:54:12.748 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:54:12.748 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:54:22.749 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:54:22.750 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:22.750 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:22.759 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:23.623 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:54:23.623 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:54:23.624 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244863 2023-10-14T00:54:23.624 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:23.625 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:54:23.625 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:54:23.625 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:54:33.632 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:54:33.633 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:33.633 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:33.638 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:34.544 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:54:34.545 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:54:34.545 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244874 2023-10-14T00:54:34.546 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:34.546 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:54:34.546 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:54:34.546 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:54:44.551 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:44.552 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:54:44.554 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:44.554 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:45.551 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:54:45.553 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:54:45.553 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244885 2023-10-14T00:54:45.554 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:45.554 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:54:45.554 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:54:45.554 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:54:55.555 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:54:55.556 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:55.556 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:55.570 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:56.466 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:54:56.467 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:54:56.467 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244896 2023-10-14T00:54:56.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:56.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:54:56.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:54:56.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:55:06.475 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:06.481 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:55:06.485 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:06.485 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:07.417 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:55:07.418 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:55:07.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244907 2023-10-14T00:55:07.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:07.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:55:07.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:55:07.420 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:55:17.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:55:17.430 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:17.431 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:17.433 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:18.302 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:55:18.302 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:55:18.303 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244918 2023-10-14T00:55:18.303 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:18.304 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:55:18.304 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:55:18.304 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:55:28.305 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:55:28.305 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:28.306 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:28.315 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:29.301 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:55:29.308 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:55:29.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244929 2023-10-14T00:55:29.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:29.309 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:55:29.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:55:29.310 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:55:39.311 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:55:39.311 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:39.327 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:39.338 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:40.260 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:55:40.260 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:55:40.261 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244940 2023-10-14T00:55:40.261 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:40.262 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:55:40.262 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:55:40.262 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:55:42.426 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.424+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:55:42.428 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.424+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:55:42.441 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:55:42.442 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:55:42.442 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:55:42.442 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:55:42.442 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:55:42.443 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:42.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:55:50.263 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:50.263 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:50.265 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:55:50.265 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:51.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:55:51.269 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:55:51.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244951 2023-10-14T00:55:51.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:51.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:55:51.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:55:51.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:55:51.435 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:55:51.432+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:55:54.706 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.704+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:55:54.707 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.704+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:55:54.707 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.704+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:55:54.707 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.704+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:55:54.716 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.708+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:55:54.716 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.708+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:55:54.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.708+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:55:54.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.708+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:55:54.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.708+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:55:54.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:55:54.712+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:56:01.278 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:01.278 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:56:01.278 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:01.278 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:01.955 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:56:01.955 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:56:01.956 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244961 2023-10-14T00:56:01.956 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:01.956 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:56:01.956 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:56:01.957 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:56:11.958 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:56:11.970 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:11.980 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:11.981 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:12.940 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:56:12.941 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:56:12.942 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244972 2023-10-14T00:56:12.942 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:12.942 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:56:12.942 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:56:12.942 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:56:22.943 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:22.944 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:56:22.944 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:22.946 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:23.850 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:56:23.850 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:56:23.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244983 2023-10-14T00:56:23.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:23.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:56:23.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:56:23.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:56:30.955 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:30.951+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:56:30.955 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:30.951+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:56:30.955 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:30.951+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:56:30.962 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:30.955+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:56:33.853 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:33.854 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:33.863 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:33.866 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:56:34.849 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:56:34.849 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:56:34.850 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697244994 2023-10-14T00:56:34.850 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:34.850 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:56:34.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:56:34.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:56:42.294 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.287+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:56:42.294 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.287+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:56:42.318 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.311+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:56:42.318 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.311+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:56:42.319 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.311+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:56:42.319 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.311+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:56:42.319 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.311+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:56:42.319 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:56:42.311+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:56:44.852 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:56:44.853 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:44.855 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:44.862 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:45.788 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:56:45.794 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:56:45.795 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245005 2023-10-14T00:56:45.795 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:45.795 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:56:45.795 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:56:45.795 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:56:55.799 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:56:55.799 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:55.799 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:55.800 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:56.608 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:56:56.609 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:56:56.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245016 2023-10-14T00:56:56.610 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:56.610 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:56:56.610 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:56:56.610 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:57:02.377 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:57:02.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:57:02.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:57:02.389 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.383+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:57:02.389 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.383+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:57:02.389 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.383+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:57:02.390 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.383+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:57:02.392 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.387+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T00:57:02.393 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:02.387+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:57:06.628 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:57:06.629 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:06.629 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:06.641 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:07.469 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:57:07.470 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:57:07.470 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245027 2023-10-14T00:57:07.470 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:07.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:57:07.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:57:07.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:57:17.472 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:57:17.473 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:17.475 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:17.475 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:18.395 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:57:18.409 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:57:18.410 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245038 2023-10-14T00:57:18.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:18.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:57:18.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:57:18.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:57:18.993 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:18.983+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:57:18.994 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:18.983+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:57:18.994 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:18.983+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:57:18.994 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:18.983+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:57:18.994 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:57:18.983+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:57:28.418 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:28.418 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:28.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:57:28.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:29.408 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:57:29.408 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:57:29.410 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245049 2023-10-14T00:57:29.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:29.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:57:29.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:57:29.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:57:39.425 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:39.430 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:39.433 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:39.438 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:57:40.300 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:57:40.300 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:57:40.302 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245060 2023-10-14T00:57:40.302 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:40.302 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:57:40.302 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:57:40.303 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:57:50.310 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:50.311 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:50.313 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:57:50.323 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:51.264 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:57:51.264 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:57:51.265 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245071 2023-10-14T00:57:51.265 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:51.265 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:57:51.265 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:57:51.266 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:58:01.267 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:01.267 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:58:01.276 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:01.277 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:02.398 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:58:02.409 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:58:02.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245082 2023-10-14T00:58:02.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:02.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:58:02.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:58:02.411 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:58:07.857 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T00:58:07.854+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T00:58:12.414 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:58:12.418 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:12.421 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:12.422 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:13.250 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:58:13.262 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:58:13.263 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245093 2023-10-14T00:58:13.264 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:13.264 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:58:13.264 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:58:13.264 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:58:16.099 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:16.098+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:58:16.099 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:16.098+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:58:16.099 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:16.098+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:58:16.100 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:16.098+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:58:16.100 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:16.098+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:58:23.270 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:58:23.271 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:23.275 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:23.276 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:24.293 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:58:24.294 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:58:24.295 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245104 2023-10-14T00:58:24.295 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:24.296 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:58:24.296 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:58:24.296 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:58:27.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:27.197+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:58:27.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:27.197+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:58:34.297 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:34.298 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:58:34.300 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:34.313 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:35.183 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:58:35.191 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:58:35.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245115 2023-10-14T00:58:35.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:35.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:58:35.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:58:35.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:58:45.196 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:58:45.196 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:45.201 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:45.205 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:46.226 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:58:46.226 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:58:46.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245126 2023-10-14T00:58:46.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:46.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:58:46.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:58:46.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:58:56.226 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:56.225+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:58:56.228 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:56.225+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:58:56.230 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:56.229+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:58:56.231 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:58:56.229+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:58:56.231 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:58:56.233 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:56.243 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:56.248 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:57.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:58:57.078 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:58:57.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245137 2023-10-14T00:58:57.079 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:57.079 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:58:57.079 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:58:57.079 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:59:05.179 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.177+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:59:05.179 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.177+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:59:05.180 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.177+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:59:05.180 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.177+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:59:05.194 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.193+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:59:05.194 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.193+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:59:05.194 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.193+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:59:05.195 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.193+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:59:05.213 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:05.205+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:59:07.078 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:59:07.079 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:07.083 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:07.084 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:08.000 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:59:08.000 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:59:08.001 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245148 2023-10-14T00:59:08.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:08.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:59:08.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:59:08.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:59:15.521 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.516+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:59:15.536 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:59:15.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:59:15.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:59:15.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:59:15.539 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:59:15.539 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:59:15.539 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:59:15.539 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.532+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:59:15.540 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:15.536+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T00:59:18.009 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:18.010 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:18.014 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:59:18.018 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:19.048 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:59:19.049 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:59:19.050 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245159 2023-10-14T00:59:19.050 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:19.051 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:59:19.051 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:59:19.051 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:59:23.578 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.572+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T00:59:23.578 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.572+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T00:59:23.586 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T00:59:23.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:59:23.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T00:59:23.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T00:59:23.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T00:59:23.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T00:59:23.588 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:59:23.588 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:23.584+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:59:24.073 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:24.068+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T00:59:24.074 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:24.068+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T00:59:24.074 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T00:59:24.068+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T00:59:29.058 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:29.059 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:29.059 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:29.061 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:59:29.900 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:59:29.900 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:59:29.901 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245169 2023-10-14T00:59:29.901 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:29.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:59:29.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:59:29.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:59:39.905 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:39.906 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:59:39.910 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:39.912 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:40.802 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:59:40.802 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:59:40.803 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245180 2023-10-14T00:59:40.803 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:40.803 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:59:40.804 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:59:40.804 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T00:59:50.805 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:50.811 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:50.819 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:50.825 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T00:59:51.890 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T00:59:51.891 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T00:59:51.893 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245191 2023-10-14T00:59:51.893 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:51.893 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T00:59:51.893 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T00:59:51.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:01.895 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:01.895 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:01.895 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:00:01.920 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:02.834 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:00:02.838 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:00:02.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245202 2023-10-14T01:00:02.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:02.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:00:02.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:00:02.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:12.841 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:00:12.842 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:12.844 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:12.852 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:13.741 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:00:13.742 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:00:13.742 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245213 2023-10-14T01:00:13.742 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:13.743 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:00:13.743 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:00:13.743 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:23.745 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:23.745 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:23.760 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:23.776 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:00:24.696 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:00:24.697 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:00:24.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245224 2023-10-14T01:00:24.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:24.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:00:24.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:00:24.699 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:34.699 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:00:34.700 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:34.701 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:34.704 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:35.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:00:35.692 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:00:35.692 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245235 2023-10-14T01:00:35.693 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:35.693 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:00:35.693 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:00:35.693 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:36.176 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:36.167+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:00:36.176 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:36.167+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:00:36.197 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:36.191+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:00:44.715 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:44.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:00:44.715 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:44.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:00:45.694 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:45.694 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:45.695 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:00:45.696 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:46.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:00:46.665 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:00:46.666 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245246 2023-10-14T01:00:46.666 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:46.666 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:00:46.666 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:00:46.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:53.535 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:00:53.535 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:00:53.535 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:00:53.535 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:00:53.536 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:00:53.536 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:00:53.536 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:00:53.536 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:00:53.536 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:00:53.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:00:53.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:00:53.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:00:53.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:00:53.537 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:00:53.538 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:53.527+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:00:56.671 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:56.672 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:56.678 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:00:56.684 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:57.524 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:00:57.524 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:00:57.525 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245257 2023-10-14T01:00:57.525 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:57.525 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:00:57.525 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:00:57.526 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:00:58.146 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.143+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:00:58.157 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.147+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:00:58.158 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.147+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:00:58.158 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.147+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:00:58.158 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.155+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:00:58.165 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.159+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:00:58.180 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:00:58.171+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:01:00.328 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.323+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:01:00.328 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.323+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:01:00.329 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.323+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:01:00.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:01:00.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:01:00.335 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:01:00.336 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:01:00.336 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:01:00.377 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:01:00.377 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:00.371+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:01:05.808 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:05.803+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:01:05.817 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:05.811+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:01:05.817 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:05.811+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:01:05.818 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:05.811+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:01:07.529 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:01:07.530 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:07.547 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:07.549 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:08.414 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:01:08.417 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:01:08.418 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245268 2023-10-14T01:01:08.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:08.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:01:08.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:01:08.419 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:01:08.847 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:08.839+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:01:08.875 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:08.871+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:01:08.875 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:08.871+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:01:08.875 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:08.871+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:01:08.875 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:08.871+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:01:18.420 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:18.421 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:18.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:01:18.427 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:19.317 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:01:19.317 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:01:19.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245279 2023-10-14T01:01:19.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:19.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:01:19.325 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:01:19.325 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:01:23.473 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:23.466+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:01:23.473 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:23.466+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:01:23.473 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:23.466+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:01:23.474 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:23.466+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:01:23.501 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:23.494+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:01:29.330 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:01:29.330 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:29.331 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:29.347 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:30.265 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:01:30.267 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:01:30.268 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245290 2023-10-14T01:01:30.268 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:30.268 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:01:30.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:01:30.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:01:40.281 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:40.289 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:01:40.290 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:40.309 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:41.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:01:41.192 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:01:41.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245301 2023-10-14T01:01:41.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:41.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:01:41.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:01:41.194 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:01:51.195 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:01:51.195 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:51.196 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:51.217 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:52.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:01:52.269 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:01:52.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245312 2023-10-14T01:01:52.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:52.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:01:52.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:01:52.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:01:57.742 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:57.742+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:01:57.743 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:57.742+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:01:57.744 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:57.742+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:01:57.744 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:01:57.742+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:02:02.274 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:02.274 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:02.290 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:02:02.291 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:03.349 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:02:03.349 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:02:03.350 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245323 2023-10-14T01:02:03.350 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:03.351 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:02:03.351 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:02:03.351 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:02:13.352 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:02:13.353 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:13.353 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:13.371 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:14.322 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:02:14.322 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:02:14.323 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245334 2023-10-14T01:02:14.323 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:14.323 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:02:14.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:02:14.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:02:24.329 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:02:24.342 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:24.343 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:24.345 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:25.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:02:25.189 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:02:25.189 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245345 2023-10-14T01:02:25.189 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:25.190 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:02:25.190 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:02:25.190 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:02:35.191 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:02:35.192 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:35.194 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:35.201 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:36.017 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:02:36.018 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:02:36.018 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245356 2023-10-14T01:02:36.019 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:36.019 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:02:36.019 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:02:36.019 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:02:37.533 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.529+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:02:37.538 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.533+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:02:37.539 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.533+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:02:37.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.545+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:02:37.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.545+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:02:37.587 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.545+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:02:37.588 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.545+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:02:37.588 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:02:37.593 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:02:37.593+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:02:46.021 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:46.021 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:46.022 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:02:46.022 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:46.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:02:46.807 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:02:46.808 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245366 2023-10-14T01:02:46.808 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:46.808 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:02:46.808 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:02:46.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:02:56.810 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:56.810 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:02:56.811 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:56.819 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:57.698 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:02:57.699 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:02:57.699 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245377 2023-10-14T01:02:57.700 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:57.700 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:02:57.700 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:02:57.700 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:03:07.718 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:03:07.719 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:07.721 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:07.721 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:08.313 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:08.308+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:03:08.314 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:08.308+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:03:08.314 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:08.308+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:03:08.646 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:03:08.649 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:03:08.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245388 2023-10-14T01:03:08.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:08.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:03:08.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:03:08.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:03:16.327 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:16.324+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:03:16.333 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:16.328+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:03:16.333 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:16.328+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:03:16.333 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:16.328+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:03:18.659 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:03:18.660 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:18.661 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:18.681 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:19.607 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:03:19.608 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:03:19.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245399 2023-10-14T01:03:19.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:19.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:03:19.609 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:03:19.610 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:03:25.707 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:03:25.704+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:03:29.611 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:29.611 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:03:29.611 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:29.622 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:30.443 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:03:30.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:03:30.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:03:30.445 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:03:30.445 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:03:30.446 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:03:30.447 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245410 2023-10-14T01:03:30.447 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:30.447 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:03:30.448 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:03:30.448 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:03:30.454 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:03:30.455 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:03:30.455 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:03:30.455 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.452+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:03:30.463 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.456+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:03:30.471 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.464+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:03:30.471 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:03:30.464+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:03:40.449 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:40.453 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:40.458 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:40.461 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:03:41.324 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:03:41.324 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:03:41.325 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245421 2023-10-14T01:03:41.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:41.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:03:41.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:03:41.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:03:51.327 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:03:51.329 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:51.331 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:51.339 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:52.183 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:03:52.185 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:03:52.186 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245432 2023-10-14T01:03:52.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:52.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:03:52.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:03:52.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:04:02.189 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:04:02.189 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:02.201 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:02.204 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:03.014 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:04:03.014 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:04:03.015 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245443 2023-10-14T01:04:03.015 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:03.015 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:04:03.016 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:04:03.016 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:04:13.017 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:04:13.017 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:13.017 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:13.021 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:13.960 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:04:14.066 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:04:14.067 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245453 2023-10-14T01:04:14.067 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:14.068 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:04:14.068 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:04:14.068 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:04:23.963 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:04:23.963 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:23.965 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:23.975 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:24.884 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:04:24.885 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:04:24.885 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245464 2023-10-14T01:04:24.886 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:24.886 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:04:24.886 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:04:24.886 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:04:25.603 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.595+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:04:25.603 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.599+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:04:25.603 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.599+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:04:25.603 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.599+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:04:25.604 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.599+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:04:25.607 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.603+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:04:25.608 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.603+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:04:25.608 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.603+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:04:25.608 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.603+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:04:25.610 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:04:25.607+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:04:26.185 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:04:26.179+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:04:34.893 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:04:34.894 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:34.894 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:34.896 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:35.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:04:35.858 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:04:35.859 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245475 2023-10-14T01:04:35.859 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:35.859 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:04:35.859 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:04:35.859 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:04:45.861 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:45.862 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:04:45.862 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:45.867 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:46.799 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:04:46.799 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:04:46.799 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245486 2023-10-14T01:04:46.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:46.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:04:46.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:04:46.800 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:04:52.227 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:04:52.223+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:04:56.801 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:04:56.801 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:56.808 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:56.822 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:57.707 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:04:57.713 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:04:57.715 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245497 2023-10-14T01:04:57.715 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:57.715 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:04:57.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:04:57.716 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:05:04.351 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:04.347+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:05:04.351 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:04.347+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:05:04.370 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:04.367+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:05:04.371 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:04.367+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:05:07.717 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:05:07.718 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:07.722 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:07.729 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:08.548 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:05:08.549 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:05:08.550 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245508 2023-10-14T01:05:08.550 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:08.550 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:05:08.550 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:05:08.551 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:05:18.565 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:05:18.565 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:18.565 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:18.575 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:19.451 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:05:19.452 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:05:19.452 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245519 2023-10-14T01:05:19.453 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:19.453 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:05:19.453 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:05:19.453 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:05:22.887 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:22.882+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:05:22.888 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:22.882+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:05:29.455 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:05:29.455 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:29.455 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:29.458 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:30.341 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:05:30.341 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:05:30.342 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245530 2023-10-14T01:05:30.342 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:30.342 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:05:30.342 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:05:30.343 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:05:40.343 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:05:40.344 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:40.346 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:40.370 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:41.177 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:05:41.178 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:05:41.178 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245541 2023-10-14T01:05:41.178 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:41.179 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:05:41.179 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:05:41.179 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:05:51.180 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:05:51.182 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:51.185 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:51.201 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:52.071 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:05:52.072 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:05:52.072 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245552 2023-10-14T01:05:52.072 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:52.073 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:05:52.073 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:05:52.073 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:05:56.360 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.358+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:05:56.360 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.358+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:05:56.360 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.358+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:05:56.374 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.370+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:05:56.374 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.370+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:05:56.374 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.370+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:05:56.374 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.370+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:05:56.383 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.382+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:05:56.392 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:05:56.390+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:06:02.074 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:06:02.074 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:02.076 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:02.080 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:03.081 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:06:03.081 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:06:03.082 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245563 2023-10-14T01:06:03.082 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:03.083 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:06:03.083 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:06:03.083 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:06:10.662 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:06:10.661+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:06:13.086 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:06:13.087 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:13.091 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:13.094 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:13.513 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:13.505+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:06:13.533 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:13.525+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:06:13.533 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:13.525+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:06:13.534 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:13.525+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:06:13.534 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:13.525+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:06:13.986 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:06:13.986 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:06:13.986 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245573 2023-10-14T01:06:13.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:13.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:06:13.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:06:13.987 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:06:23.989 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:23.993 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:23.995 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:24.012 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:06:24.808 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:06:24.808 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:06:24.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245584 2023-10-14T01:06:24.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:24.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:06:24.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:06:24.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:06:29.518 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:06:29.473+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:06:34.810 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:34.812 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:34.818 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:34.821 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:06:35.653 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:06:35.653 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:06:35.654 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245595 2023-10-14T01:06:35.654 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:35.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:06:35.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:06:35.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:06:45.656 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:06:45.657 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:45.657 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:45.661 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:46.585 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:06:46.586 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:06:46.587 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245606 2023-10-14T01:06:46.587 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:46.587 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:06:46.587 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:06:46.588 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:06:50.616 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:06:50.616 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:06:50.616 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:06:50.616 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:06:50.616 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:06:50.617 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:06:50.617 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:06:50.617 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:06:50.617 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:06:50.617 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:06:50.618 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:06:50.618 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:06:50.618 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:06:50.618 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:06:50.618 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:06:50.613+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:06:56.589 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:06:56.593 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:56.593 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:56.603 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:57.427 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:06:57.428 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:06:57.428 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245617 2023-10-14T01:06:57.429 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:57.429 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:06:57.429 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:06:57.429 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:07:07.430 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:07:07.431 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:07:07.431 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:07:07.433 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:07:08.403 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:07:08.403 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:07:08.404 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245628 2023-10-14T01:07:08.404 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:07:08.405 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:07:08.405 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:07:08.405 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:07:18.218 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:18.216+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:07:18.230 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:18.228+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:07:18.406 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:07:18.408 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:07:18.413 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:07:18.428 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:07:19.366 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:07:19.369 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:07:19.370 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245639 2023-10-14T01:07:19.371 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:07:19.371 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ -n 172.21.15.81:0/539853599 ]] 2023-10-14T01:07:19.372 INFO:tasks.workunit.client.0.smithi081.stderr:+ [[ 172.21.15.81:0/539853599 != \1\7\2\.\2\1\.\1\5\.\8\1\:\0\/\5\3\9\8\5\3\5\9\9 ]] 2023-10-14T01:07:19.372 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:07:29.382 INFO:tasks.workunit.client.0.smithi081.stderr:++ ceph mgr dump 2023-10-14T01:07:29.386 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:07:29.391 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:07:29.393 INFO:tasks.workunit.client.0.smithi081.stderr:++ jq '.active_clients[]' 2023-10-14T01:07:29.443 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:29.436+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:07:29.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:29.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:07:29.444 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:29.440+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:07:30.290 INFO:tasks.workunit.client.0.smithi081.stderr:+ CLIENT_ADDR=172.21.15.81:0/539853599 2023-10-14T01:07:30.290 INFO:tasks.workunit.client.0.smithi081.stderr:++ date +%s 2023-10-14T01:07:30.291 INFO:tasks.workunit.client.0.smithi081.stderr:+ CURRENT_TIME=1697245650 2023-10-14T01:07:30.291 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:07:30.291 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i = 1 )) 2023-10-14T01:07:30.292 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:07:30.292 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:30.497 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:07:30.492+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:30.497 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:30.513 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:07:31.887 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:31.884+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:07:31.887 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:07:31.884+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:07:40.514 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:07:40.515 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:07:40.515 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:40.596 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:07:40.592+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:40.597 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:40.604 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:07:50.606 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:07:50.606 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:07:50.606 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:50.679 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:07:50.676+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:50.680 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:50.693 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:08:00.694 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:08:00.694 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:08:00.695 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:00.763 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:08:00.760+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:00.763 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:00.780 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:08:10.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:08:10.782 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:08:10.783 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:10.876 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:08:10.871+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:10.880 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:10.892 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:08:20.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:08:20.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:08:20.894 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:20.982 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:08:20.979+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:20.983 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:20.992 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:08:30.993 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:08:30.993 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:08:30.994 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:31.078 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:08:31.075+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:31.079 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:31.088 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:08:41.090 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:08:41.090 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:08:41.090 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:41.156 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:08:41.151+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:41.157 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:41.162 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:08:51.164 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:08:51.164 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:08:51.164 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:51.264 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:08:51.259+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:51.264 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:51.269 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:09:01.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:09:01.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:09:01.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:01.350 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:09:01.347+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:01.350 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:01.363 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:09:06.357 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:09:06.350+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:09:06.358 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:09:06.350+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:09:06.358 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:09:06.350+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:09:11.365 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:09:11.366 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:09:11.366 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:11.486 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:09:11.482+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:11.487 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:11.499 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:09:21.500 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:09:21.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:09:21.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:21.568 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:09:21.562+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:21.569 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:21.579 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:09:31.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:09:31.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:09:31.581 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:31.658 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:09:31.654+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:31.660 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:31.677 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:09:41.678 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:09:41.679 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:09:41.679 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:41.832 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:09:41.830+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:41.838 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:41.853 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:09:51.854 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:09:51.855 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:09:51.855 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:51.955 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:09:51.954+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:51.955 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:51.964 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:10:01.966 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:10:01.966 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:10:01.966 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:10:02.059 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:10:02.058+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:02.060 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:02.075 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:10:11.197 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:10:11.197+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:10:11.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:10:11.197+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:10:11.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:10:11.201+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:10:12.077 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:10:12.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:10:12.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:10:12.217 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:10:12.217+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:12.218 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:12.225 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:10:22.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:10:22.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:10:22.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:10:22.321 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:10:22.321+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:22.321 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:22.326 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:10:32.327 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:10:32.327 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:10:32.327 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:10:32.444 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:10:32.441+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:32.445 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:32.475 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:10:42.477 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:10:42.477 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:10:42.477 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:10:42.590 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:10:42.589+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:42.591 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:42.604 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:10:47.452 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:10:47.449+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:10:47.452 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:10:47.453+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:10:52.612 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:10:52.612 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:10:52.612 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:10:52.707 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:10:52.705+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:52.707 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:52.717 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:11:02.723 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:11:02.724 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:11:02.724 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:11:02.827 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:11:02.824+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:11:02.831 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:11:02.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:11:12.843 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:11:12.843 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:11:12.844 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:11:12.934 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:11:12.932+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:11:12.934 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:11:12.939 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:11:22.228 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:11:22.224+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:11:22.235 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:11:22.232+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:11:22.940 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:11:22.941 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:11:22.941 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:11:23.009 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:11:23.008+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:11:23.010 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:11:23.018 INFO:tasks.workunit.client.0.smithi081.stderr:+ sleep 10 2023-10-14T01:11:33.025 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i++ )) 2023-10-14T01:11:33.025 INFO:tasks.workunit.client.0.smithi081.stderr:+ (( i <= 24 )) 2023-10-14T01:11:33.025 INFO:tasks.workunit.client.0.smithi081.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-14T01:11:33.026 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep 'every 2m' 2023-10-14T01:11:33.106 INFO:tasks.ceph.mgr.x.smithi081.stderr:2023-10-14T01:11:33.104+0000 7f397cb7e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:11:33.107 INFO:tasks.workunit.client.0.smithi081.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:11:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T01:11:33.141 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-14T01:11:33.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-14T01:11:33.947 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-14T01:11:34.013 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=51a70f87f6174c1ba21df61da8d90058 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-14T01:11:34.016 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T01:11:34.063 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-14T01:11:34.073 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T01:11:34.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:34.624 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:11:34.625 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:11:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2108,"stamp":"2023-10-14T01:11:33.406666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26809609957,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1838664,"num_read_kb":7128490,"num_write":1802408,"num_write_kb":7208691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83511,"ondisk_log_size":83511,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17783424,"kb_used_data":14414888,"kb_used_omap":2016,"kb_used_meta":3366495,"kb_avail":265332096,"statfs":{"total":289910292480,"available":271700066304,"internally_reserved":0,"allocated":14760845312,"data_stored":23940638444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3447291435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,5,8,13,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25948160,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6772,"num_read_kb":26988,"num_write":6666,"num_write_kb":26664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001777"},"pg_stats":[{"pgid":"2.7","version":"118'6006","reported_seq":8949,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.976937+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T01:11:31.976937+0000","last_peered":"2023-10-14T01:11:31.976937+0000","last_clean":"2023-10-14T01:11:31.976937+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:11:31.976937+0000","last_undegraded":"2023-10-14T01:11:31.976937+0000","last_fullsized":"2023-10-14T01:11:31.976937+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:17:51.271717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'8946","reported_seq":14546,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.986595+0000","last_change":"2023-10-14T00:05:17.004867+0000","last_active":"2023-10-14T01:11:31.986595+0000","last_peered":"2023-10-14T01:11:31.986595+0000","last_clean":"2023-10-14T01:11:31.986595+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:11:31.986595+0000","last_undegraded":"2023-10-14T01:11:31.986595+0000","last_fullsized":"2023-10-14T01:11:31.986595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8946,"log_dups_size":0,"ondisk_log_size":8946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:48:27.138266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"118'5976","reported_seq":8719,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.976869+0000","last_change":"2023-10-14T00:05:17.002899+0000","last_active":"2023-10-14T01:11:31.976869+0000","last_peered":"2023-10-14T01:11:31.976869+0000","last_clean":"2023-10-14T01:11:31.976869+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:11:31.976869+0000","last_undegraded":"2023-10-14T01:11:31.976869+0000","last_fullsized":"2023-10-14T01:11:31.976869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:43:16.014296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"118'14746","reported_seq":22097,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.002279+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T01:11:32.002279+0000","last_peered":"2023-10-14T01:11:32.002279+0000","last_clean":"2023-10-14T01:11:32.002279+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:11:32.002279+0000","last_undegraded":"2023-10-14T01:11:32.002279+0000","last_fullsized":"2023-10-14T01:11:32.002279+0000","mapping_epoch":12,"log_start":"77'4700","ondisk_log_start":"77'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:37:32.329836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"118'467370","reported_seq":930515,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.365238+0000","last_change":"2023-10-14T00:20:08.229338+0000","last_active":"2023-10-14T01:11:32.365238+0000","last_peered":"2023-10-14T01:11:32.365238+0000","last_clean":"2023-10-14T01:11:32.365238+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:11:32.365238+0000","last_undegraded":"2023-10-14T01:11:32.365238+0000","last_fullsized":"2023-10-14T01:11:32.365238+0000","mapping_epoch":15,"log_start":"117'457333","ondisk_log_start":"117'457333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:31:10.509128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6924124160,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":465483,"num_read_kb":1831436,"num_write":464850,"num_write_kb":1859400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"118'3115","reported_seq":4662,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:30.182057+0000","last_change":"2023-10-14T00:05:17.000382+0000","last_active":"2023-10-14T01:11:30.182057+0000","last_peered":"2023-10-14T01:11:30.182057+0000","last_clean":"2023-10-14T01:11:30.182057+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:11:30.182057+0000","last_undegraded":"2023-10-14T01:11:30.182057+0000","last_fullsized":"2023-10-14T01:11:30.182057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:02:11.052958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"118'443993","reported_seq":884201,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.207175+0000","last_change":"2023-10-14T00:20:08.199640+0000","last_active":"2023-10-14T01:11:32.207175+0000","last_peered":"2023-10-14T01:11:32.207175+0000","last_clean":"2023-10-14T01:11:32.207175+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:11:32.207175+0000","last_undegraded":"2023-10-14T01:11:32.207175+0000","last_fullsized":"2023-10-14T01:11:32.207175+0000","mapping_epoch":15,"log_start":"117'433923","ondisk_log_start":"117'433923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:08:30.316718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6576390144,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":442428,"num_read_kb":1740172,"num_write":441597,"num_write_kb":1766388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'84","reported_seq":2776,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:28.722632+0000","last_change":"2023-10-14T00:05:16.999347+0000","last_active":"2023-10-14T01:11:28.722632+0000","last_peered":"2023-10-14T01:11:28.722632+0000","last_clean":"2023-10-14T01:11:28.722632+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:11:28.722632+0000","last_undegraded":"2023-10-14T01:11:28.722632+0000","last_fullsized":"2023-10-14T01:11:28.722632+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:57:01.029050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"118'461456","reported_seq":918884,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.358966+0000","last_change":"2023-10-14T00:20:08.294701+0000","last_active":"2023-10-14T01:11:32.358966+0000","last_peered":"2023-10-14T01:11:32.358966+0000","last_clean":"2023-10-14T01:11:32.358966+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:11:32.358966+0000","last_undegraded":"2023-10-14T01:11:32.358966+0000","last_fullsized":"2023-10-14T01:11:32.358966+0000","mapping_epoch":15,"log_start":"117'451427","ondisk_log_start":"117'451427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:26:25.273720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6822436864,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":459706,"num_read_kb":1808476,"num_write":458996,"num_write_kb":1835984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'119","reported_seq":2925,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:28.495293+0000","last_change":"2023-10-14T00:05:17.001059+0000","last_active":"2023-10-14T01:11:28.495293+0000","last_peered":"2023-10-14T01:11:28.495293+0000","last_clean":"2023-10-14T01:11:28.495293+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:11:28.495293+0000","last_undegraded":"2023-10-14T01:11:28.495293+0000","last_fullsized":"2023-10-14T01:11:28.495293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:16:12.406474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"118'438521","reported_seq":873650,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.199373+0000","last_change":"2023-10-14T00:20:08.207432+0000","last_active":"2023-10-14T01:11:32.199373+0000","last_peered":"2023-10-14T01:11:32.199373+0000","last_clean":"2023-10-14T01:11:32.199373+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:11:32.199373+0000","last_undegraded":"2023-10-14T01:11:32.199373+0000","last_fullsized":"2023-10-14T01:11:32.199373+0000","mapping_epoch":15,"log_start":"117'428422","ondisk_log_start":"117'428422","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:59:56.456702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6486065171,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":437299,"num_read_kb":1719600,"num_write":436169,"num_write_kb":1744670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'8887","reported_seq":12903,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.995423+0000","last_change":"2023-10-14T00:05:17.005268+0000","last_active":"2023-10-14T01:11:31.995423+0000","last_peered":"2023-10-14T01:11:31.995423+0000","last_clean":"2023-10-14T01:11:31.995423+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:11:31.995423+0000","last_undegraded":"2023-10-14T01:11:31.995423+0000","last_fullsized":"2023-10-14T01:11:31.995423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:44:40.852686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":277,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-14T01:09:36.554984+0000","last_change":"2023-10-14T00:05:12.384839+0000","last_active":"2023-10-14T01:09:36.554984+0000","last_peered":"2023-10-14T01:09:36.554984+0000","last_clean":"2023-10-14T01:09:36.554984+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:09:36.554984+0000","last_undegraded":"2023-10-14T01:09:36.554984+0000","last_fullsized":"2023-10-14T01:09:36.554984+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_clean_scrub_stamp":"2023-10-14T00:05:10.931569+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:10:36.076607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26809016339,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1804916,"num_read_kb":7099684,"num_write":1801612,"num_write_kb":7206442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14757928960,"data_stored":23940292646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40235,"ondisk_log_size":40235,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43179,"ondisk_log_size":43179,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8130192,"kb_used_data":7205256,"kb_used_omap":1043,"kb_used_meta":923884,"kb_avail":86241648,"statfs":{"total":96636764160,"available":88311447552,"internally_reserved":0,"allocated":7378182144,"data_stored":11968595821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":946057463},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.17999999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.198,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":0.39900000000000002,"5min":27.538,"15min":27.538},"last":0.191}]},{"osd":1,"last update":"Sat Oct 14 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.17899999999999999,"15min":0.17499999999999999},"max":{"1min":3.298,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.42299999999999999,"15min":0.49099999999999999},"min":{"1min":0.24099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.339,"5min":3.339,"15min":12.361000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4588448,"kb_used_data":3512600,"kb_used_omap":472,"kb_used_meta":1075367,"kb_avail":89783392,"statfs":{"total":96636764160,"available":91938193408,"internally_reserved":0,"allocated":3596902400,"data_stored":5837390833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1101176527},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.42899999999999999},"min":{"1min":0.182,"5min":0.17100000000000001,"15min":0.16400000000000001},"max":{"1min":3.2570000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.59999999999999998,"15min":0.52400000000000002},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":4.4969999999999999,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:10:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.8160000000000001,"5min":15.891,"15min":15.891},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.48099999999999998,"15min":0.57599999999999996},"min":{"1min":0.221,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":1.925,"5min":5.6699999999999999,"15min":9.673},"last":0.23100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739166,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064784,"kb_used_data":3697032,"kb_used_omap":500,"kb_used_meta":1367243,"kb_avail":89307056,"statfs":{"total":96636764160,"available":91450425344,"internally_reserved":0,"allocated":3785760768,"data_stored":6134651790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1400057445},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.45700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.16800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.47699999999999998,"15min":0.53400000000000003},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":5.1909999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.214},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.53900000000000003},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3784077312,"data_stored":6133805056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3595325440,"data_stored":5836779539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7378526208,"data_stored":11969708051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:34.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:35.147 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:11:35.148 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:11:35.189 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2108,"stamp":"2023-10-14T01:11:33.406666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26809609957,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1838664,"num_read_kb":7128490,"num_write":1802408,"num_write_kb":7208691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83511,"ondisk_log_size":83511,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17783424,"kb_used_data":14414888,"kb_used_omap":2016,"kb_used_meta":3366495,"kb_avail":265332096,"statfs":{"total":289910292480,"available":271700066304,"internally_reserved":0,"allocated":14760845312,"data_stored":23940638444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3447291435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,5,8,13,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25948160,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6772,"num_read_kb":26988,"num_write":6666,"num_write_kb":26664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001777"},"pg_stats":[{"pgid":"2.7","version":"118'6006","reported_seq":8949,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.976937+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T01:11:31.976937+0000","last_peered":"2023-10-14T01:11:31.976937+0000","last_clean":"2023-10-14T01:11:31.976937+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:11:31.976937+0000","last_undegraded":"2023-10-14T01:11:31.976937+0000","last_fullsized":"2023-10-14T01:11:31.976937+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:17:51.271717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'8946","reported_seq":14546,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.986595+0000","last_change":"2023-10-14T00:05:17.004867+0000","last_active":"2023-10-14T01:11:31.986595+0000","last_peered":"2023-10-14T01:11:31.986595+0000","last_clean":"2023-10-14T01:11:31.986595+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:11:31.986595+0000","last_undegraded":"2023-10-14T01:11:31.986595+0000","last_fullsized":"2023-10-14T01:11:31.986595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8946,"log_dups_size":0,"ondisk_log_size":8946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:48:27.138266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"118'5976","reported_seq":8719,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.976869+0000","last_change":"2023-10-14T00:05:17.002899+0000","last_active":"2023-10-14T01:11:31.976869+0000","last_peered":"2023-10-14T01:11:31.976869+0000","last_clean":"2023-10-14T01:11:31.976869+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:11:31.976869+0000","last_undegraded":"2023-10-14T01:11:31.976869+0000","last_fullsized":"2023-10-14T01:11:31.976869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:43:16.014296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"118'14746","reported_seq":22097,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.002279+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T01:11:32.002279+0000","last_peered":"2023-10-14T01:11:32.002279+0000","last_clean":"2023-10-14T01:11:32.002279+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:11:32.002279+0000","last_undegraded":"2023-10-14T01:11:32.002279+0000","last_fullsized":"2023-10-14T01:11:32.002279+0000","mapping_epoch":12,"log_start":"77'4700","ondisk_log_start":"77'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:37:32.329836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"118'467370","reported_seq":930515,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.365238+0000","last_change":"2023-10-14T00:20:08.229338+0000","last_active":"2023-10-14T01:11:32.365238+0000","last_peered":"2023-10-14T01:11:32.365238+0000","last_clean":"2023-10-14T01:11:32.365238+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:11:32.365238+0000","last_undegraded":"2023-10-14T01:11:32.365238+0000","last_fullsized":"2023-10-14T01:11:32.365238+0000","mapping_epoch":15,"log_start":"117'457333","ondisk_log_start":"117'457333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:31:10.509128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6924124160,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":465483,"num_read_kb":1831436,"num_write":464850,"num_write_kb":1859400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"118'3115","reported_seq":4662,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:30.182057+0000","last_change":"2023-10-14T00:05:17.000382+0000","last_active":"2023-10-14T01:11:30.182057+0000","last_peered":"2023-10-14T01:11:30.182057+0000","last_clean":"2023-10-14T01:11:30.182057+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:11:30.182057+0000","last_undegraded":"2023-10-14T01:11:30.182057+0000","last_fullsized":"2023-10-14T01:11:30.182057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:02:11.052958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"118'443993","reported_seq":884201,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.207175+0000","last_change":"2023-10-14T00:20:08.199640+0000","last_active":"2023-10-14T01:11:32.207175+0000","last_peered":"2023-10-14T01:11:32.207175+0000","last_clean":"2023-10-14T01:11:32.207175+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:11:32.207175+0000","last_undegraded":"2023-10-14T01:11:32.207175+0000","last_fullsized":"2023-10-14T01:11:32.207175+0000","mapping_epoch":15,"log_start":"117'433923","ondisk_log_start":"117'433923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:08:30.316718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6576390144,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":442428,"num_read_kb":1740172,"num_write":441597,"num_write_kb":1766388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'84","reported_seq":2776,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:28.722632+0000","last_change":"2023-10-14T00:05:16.999347+0000","last_active":"2023-10-14T01:11:28.722632+0000","last_peered":"2023-10-14T01:11:28.722632+0000","last_clean":"2023-10-14T01:11:28.722632+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:11:28.722632+0000","last_undegraded":"2023-10-14T01:11:28.722632+0000","last_fullsized":"2023-10-14T01:11:28.722632+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:57:01.029050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"118'461456","reported_seq":918884,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.358966+0000","last_change":"2023-10-14T00:20:08.294701+0000","last_active":"2023-10-14T01:11:32.358966+0000","last_peered":"2023-10-14T01:11:32.358966+0000","last_clean":"2023-10-14T01:11:32.358966+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:11:32.358966+0000","last_undegraded":"2023-10-14T01:11:32.358966+0000","last_fullsized":"2023-10-14T01:11:32.358966+0000","mapping_epoch":15,"log_start":"117'451427","ondisk_log_start":"117'451427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:26:25.273720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6822436864,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":459706,"num_read_kb":1808476,"num_write":458996,"num_write_kb":1835984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'119","reported_seq":2925,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:28.495293+0000","last_change":"2023-10-14T00:05:17.001059+0000","last_active":"2023-10-14T01:11:28.495293+0000","last_peered":"2023-10-14T01:11:28.495293+0000","last_clean":"2023-10-14T01:11:28.495293+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:11:28.495293+0000","last_undegraded":"2023-10-14T01:11:28.495293+0000","last_fullsized":"2023-10-14T01:11:28.495293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:16:12.406474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"118'438521","reported_seq":873650,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.199373+0000","last_change":"2023-10-14T00:20:08.207432+0000","last_active":"2023-10-14T01:11:32.199373+0000","last_peered":"2023-10-14T01:11:32.199373+0000","last_clean":"2023-10-14T01:11:32.199373+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:11:32.199373+0000","last_undegraded":"2023-10-14T01:11:32.199373+0000","last_fullsized":"2023-10-14T01:11:32.199373+0000","mapping_epoch":15,"log_start":"117'428422","ondisk_log_start":"117'428422","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:59:56.456702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6486065171,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":437299,"num_read_kb":1719600,"num_write":436169,"num_write_kb":1744670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'8887","reported_seq":12903,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.995423+0000","last_change":"2023-10-14T00:05:17.005268+0000","last_active":"2023-10-14T01:11:31.995423+0000","last_peered":"2023-10-14T01:11:31.995423+0000","last_clean":"2023-10-14T01:11:31.995423+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:11:31.995423+0000","last_undegraded":"2023-10-14T01:11:31.995423+0000","last_fullsized":"2023-10-14T01:11:31.995423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:44:40.852686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":277,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-14T01:09:36.554984+0000","last_change":"2023-10-14T00:05:12.384839+0000","last_active":"2023-10-14T01:09:36.554984+0000","last_peered":"2023-10-14T01:09:36.554984+0000","last_clean":"2023-10-14T01:09:36.554984+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:09:36.554984+0000","last_undegraded":"2023-10-14T01:09:36.554984+0000","last_fullsized":"2023-10-14T01:09:36.554984+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_clean_scrub_stamp":"2023-10-14T00:05:10.931569+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:10:36.076607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26809016339,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1804916,"num_read_kb":7099684,"num_write":1801612,"num_write_kb":7206442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14757928960,"data_stored":23940292646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40235,"ondisk_log_size":40235,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43179,"ondisk_log_size":43179,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8130192,"kb_used_data":7205256,"kb_used_omap":1043,"kb_used_meta":923884,"kb_avail":86241648,"statfs":{"total":96636764160,"available":88311447552,"internally_reserved":0,"allocated":7378182144,"data_stored":11968595821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":946057463},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.17999999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.198,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":0.39900000000000002,"5min":27.538,"15min":27.538},"last":0.191}]},{"osd":1,"last update":"Sat Oct 14 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.17899999999999999,"15min":0.17499999999999999},"max":{"1min":3.298,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.42299999999999999,"15min":0.49099999999999999},"min":{"1min":0.24099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.339,"5min":3.339,"15min":12.361000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4588448,"kb_used_data":3512600,"kb_used_omap":472,"kb_used_meta":1075367,"kb_avail":89783392,"statfs":{"total":96636764160,"available":91938193408,"internally_reserved":0,"allocated":3596902400,"data_stored":5837390833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1101176527},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.42899999999999999},"min":{"1min":0.182,"5min":0.17100000000000001,"15min":0.16400000000000001},"max":{"1min":3.2570000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.59999999999999998,"15min":0.52400000000000002},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":4.4969999999999999,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:10:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.8160000000000001,"5min":15.891,"15min":15.891},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.48099999999999998,"15min":0.57599999999999996},"min":{"1min":0.221,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":1.925,"5min":5.6699999999999999,"15min":9.673},"last":0.23100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739166,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064784,"kb_used_data":3697032,"kb_used_omap":500,"kb_used_meta":1367243,"kb_avail":89307056,"statfs":{"total":96636764160,"available":91450425344,"internally_reserved":0,"allocated":3785760768,"data_stored":6134651790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1400057445},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.45700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.16800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.47699999999999998,"15min":0.53400000000000003},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":5.1909999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.214},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.53900000000000003},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3784077312,"data_stored":6133805056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3595325440,"data_stored":5836779539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7378526208,"data_stored":11969708051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:35.190 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T01:11:35.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:11:35.730 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:11:35.778 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2109,"stamp":"2023-10-14T01:11:35.407038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26809609957,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1838664,"num_read_kb":7128490,"num_write":1802408,"num_write_kb":7208691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83511,"ondisk_log_size":83511,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17783424,"kb_used_data":14414888,"kb_used_omap":2016,"kb_used_meta":3366495,"kb_avail":265332096,"statfs":{"total":289910292480,"available":271700066304,"internally_reserved":0,"allocated":14760845312,"data_stored":23940638444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3447291435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,5,8,13,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17227776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4538,"num_read_kb":18080,"num_write":4423,"num_write_kb":17692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001840"},"pg_stats":[{"pgid":"2.7","version":"118'6006","reported_seq":8949,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.976937+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T01:11:31.976937+0000","last_peered":"2023-10-14T01:11:31.976937+0000","last_clean":"2023-10-14T01:11:31.976937+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:11:31.976937+0000","last_undegraded":"2023-10-14T01:11:31.976937+0000","last_fullsized":"2023-10-14T01:11:31.976937+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:17:51.271717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'8946","reported_seq":14546,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.986595+0000","last_change":"2023-10-14T00:05:17.004867+0000","last_active":"2023-10-14T01:11:31.986595+0000","last_peered":"2023-10-14T01:11:31.986595+0000","last_clean":"2023-10-14T01:11:31.986595+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:11:31.986595+0000","last_undegraded":"2023-10-14T01:11:31.986595+0000","last_fullsized":"2023-10-14T01:11:31.986595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8946,"log_dups_size":0,"ondisk_log_size":8946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:48:27.138266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"118'5976","reported_seq":8719,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.976869+0000","last_change":"2023-10-14T00:05:17.002899+0000","last_active":"2023-10-14T01:11:31.976869+0000","last_peered":"2023-10-14T01:11:31.976869+0000","last_clean":"2023-10-14T01:11:31.976869+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:11:31.976869+0000","last_undegraded":"2023-10-14T01:11:31.976869+0000","last_fullsized":"2023-10-14T01:11:31.976869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:43:16.014296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"118'14746","reported_seq":22097,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.002279+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T01:11:32.002279+0000","last_peered":"2023-10-14T01:11:32.002279+0000","last_clean":"2023-10-14T01:11:32.002279+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:11:32.002279+0000","last_undegraded":"2023-10-14T01:11:32.002279+0000","last_fullsized":"2023-10-14T01:11:32.002279+0000","mapping_epoch":12,"log_start":"77'4700","ondisk_log_start":"77'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:37:32.329836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"118'467370","reported_seq":930515,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.365238+0000","last_change":"2023-10-14T00:20:08.229338+0000","last_active":"2023-10-14T01:11:32.365238+0000","last_peered":"2023-10-14T01:11:32.365238+0000","last_clean":"2023-10-14T01:11:32.365238+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:11:32.365238+0000","last_undegraded":"2023-10-14T01:11:32.365238+0000","last_fullsized":"2023-10-14T01:11:32.365238+0000","mapping_epoch":15,"log_start":"117'457333","ondisk_log_start":"117'457333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:31:10.509128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6924124160,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":465483,"num_read_kb":1831436,"num_write":464850,"num_write_kb":1859400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"118'3115","reported_seq":4662,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:30.182057+0000","last_change":"2023-10-14T00:05:17.000382+0000","last_active":"2023-10-14T01:11:30.182057+0000","last_peered":"2023-10-14T01:11:30.182057+0000","last_clean":"2023-10-14T01:11:30.182057+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:11:30.182057+0000","last_undegraded":"2023-10-14T01:11:30.182057+0000","last_fullsized":"2023-10-14T01:11:30.182057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:02:11.052958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"118'443993","reported_seq":884201,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.207175+0000","last_change":"2023-10-14T00:20:08.199640+0000","last_active":"2023-10-14T01:11:32.207175+0000","last_peered":"2023-10-14T01:11:32.207175+0000","last_clean":"2023-10-14T01:11:32.207175+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:11:32.207175+0000","last_undegraded":"2023-10-14T01:11:32.207175+0000","last_fullsized":"2023-10-14T01:11:32.207175+0000","mapping_epoch":15,"log_start":"117'433923","ondisk_log_start":"117'433923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:08:30.316718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6576390144,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":442428,"num_read_kb":1740172,"num_write":441597,"num_write_kb":1766388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'84","reported_seq":2776,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:28.722632+0000","last_change":"2023-10-14T00:05:16.999347+0000","last_active":"2023-10-14T01:11:28.722632+0000","last_peered":"2023-10-14T01:11:28.722632+0000","last_clean":"2023-10-14T01:11:28.722632+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:11:28.722632+0000","last_undegraded":"2023-10-14T01:11:28.722632+0000","last_fullsized":"2023-10-14T01:11:28.722632+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:57:01.029050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"118'461456","reported_seq":918884,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.358966+0000","last_change":"2023-10-14T00:20:08.294701+0000","last_active":"2023-10-14T01:11:32.358966+0000","last_peered":"2023-10-14T01:11:32.358966+0000","last_clean":"2023-10-14T01:11:32.358966+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:11:32.358966+0000","last_undegraded":"2023-10-14T01:11:32.358966+0000","last_fullsized":"2023-10-14T01:11:32.358966+0000","mapping_epoch":15,"log_start":"117'451427","ondisk_log_start":"117'451427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:26:25.273720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6822436864,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":459706,"num_read_kb":1808476,"num_write":458996,"num_write_kb":1835984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'119","reported_seq":2925,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:28.495293+0000","last_change":"2023-10-14T00:05:17.001059+0000","last_active":"2023-10-14T01:11:28.495293+0000","last_peered":"2023-10-14T01:11:28.495293+0000","last_clean":"2023-10-14T01:11:28.495293+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:11:28.495293+0000","last_undegraded":"2023-10-14T01:11:28.495293+0000","last_fullsized":"2023-10-14T01:11:28.495293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:16:12.406474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"118'438521","reported_seq":873650,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.199373+0000","last_change":"2023-10-14T00:20:08.207432+0000","last_active":"2023-10-14T01:11:32.199373+0000","last_peered":"2023-10-14T01:11:32.199373+0000","last_clean":"2023-10-14T01:11:32.199373+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:11:32.199373+0000","last_undegraded":"2023-10-14T01:11:32.199373+0000","last_fullsized":"2023-10-14T01:11:32.199373+0000","mapping_epoch":15,"log_start":"117'428422","ondisk_log_start":"117'428422","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:59:56.456702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6486065171,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":437299,"num_read_kb":1719600,"num_write":436169,"num_write_kb":1744670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'8887","reported_seq":12903,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:31.995423+0000","last_change":"2023-10-14T00:05:17.005268+0000","last_active":"2023-10-14T01:11:31.995423+0000","last_peered":"2023-10-14T01:11:31.995423+0000","last_clean":"2023-10-14T01:11:31.995423+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:11:31.995423+0000","last_undegraded":"2023-10-14T01:11:31.995423+0000","last_fullsized":"2023-10-14T01:11:31.995423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:44:40.852686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":277,"reported_epoch":117,"state":"active+clean","last_fresh":"2023-10-14T01:09:36.554984+0000","last_change":"2023-10-14T00:05:12.384839+0000","last_active":"2023-10-14T01:09:36.554984+0000","last_peered":"2023-10-14T01:09:36.554984+0000","last_clean":"2023-10-14T01:09:36.554984+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:09:36.554984+0000","last_undegraded":"2023-10-14T01:09:36.554984+0000","last_fullsized":"2023-10-14T01:09:36.554984+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_clean_scrub_stamp":"2023-10-14T00:05:10.931569+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:10:36.076607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26809016339,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1804916,"num_read_kb":7099684,"num_write":1801612,"num_write_kb":7206442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14757928960,"data_stored":23940292646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40235,"ondisk_log_size":40235,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43179,"ondisk_log_size":43179,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8130192,"kb_used_data":7205256,"kb_used_omap":1043,"kb_used_meta":923884,"kb_avail":86241648,"statfs":{"total":96636764160,"available":88311447552,"internally_reserved":0,"allocated":7378182144,"data_stored":11968595821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":946057463},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.17999999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.198,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":0.39900000000000002,"5min":27.538,"15min":27.538},"last":0.191}]},{"osd":1,"last update":"Sat Oct 14 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.17899999999999999,"15min":0.17499999999999999},"max":{"1min":3.298,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.42299999999999999,"15min":0.49099999999999999},"min":{"1min":0.24099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.339,"5min":3.339,"15min":12.361000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4588448,"kb_used_data":3512600,"kb_used_omap":472,"kb_used_meta":1075367,"kb_avail":89783392,"statfs":{"total":96636764160,"available":91938193408,"internally_reserved":0,"allocated":3596902400,"data_stored":5837390833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1101176527},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.42899999999999999},"min":{"1min":0.182,"5min":0.17100000000000001,"15min":0.16400000000000001},"max":{"1min":3.2570000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.59999999999999998,"15min":0.52400000000000002},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":4.4969999999999999,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:10:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.18099999999999999},"max":{"1min":1.8160000000000001,"5min":15.891,"15min":15.891},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.48099999999999998,"15min":0.57599999999999996},"min":{"1min":0.221,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":1.925,"5min":5.6699999999999999,"15min":9.673},"last":0.23100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739166,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064784,"kb_used_data":3697032,"kb_used_omap":500,"kb_used_meta":1367243,"kb_avail":89307056,"statfs":{"total":96636764160,"available":91450425344,"internally_reserved":0,"allocated":3785760768,"data_stored":6134651790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1400057445},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.45700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.16800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.47699999999999998,"15min":0.53400000000000003},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":5.1909999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.214},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.53900000000000003},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3784077312,"data_stored":6133805056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3595325440,"data_stored":5836779539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7378526208,"data_stored":11969708051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:35.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T01:11:36.306 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:11:36.307 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"fsid":"e624b2c7-92dd-4594-9e39-e4d11116e328","created":"2023-10-14T00:04:59.965973+0000","modified":"2023-10-14T01:10:41.146413+0000","last_up_change":"2023-10-14T00:05:09.785799+0000","last_in_change":"2023-10-14T00:05:00.869422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:05:10.615039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-14T00:05:15.306730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-14T00:05:21.020212+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":63,"snap_epoch":92,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"87b96572-35f6-4cb9-a0e4-c63532c7b020","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6801","nonce":290843498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6803","nonce":290843498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6807","nonce":290843498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":290843498},{"type":"v1","addr":"172.21.15.81:6805","nonce":290843498}]},"public_addr":"172.21.15.81:6801/290843498","cluster_addr":"172.21.15.81:6803/290843498","heartbeat_back_addr":"172.21.15.81:6807/290843498","heartbeat_front_addr":"172.21.15.81:6805/290843498","state":["exists","up"]},{"osd":1,"uuid":"d0c8a7bb-27b3-4a47-af27-af7a89c05721","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6817","nonce":3262473869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6819","nonce":3262473869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6823","nonce":3262473869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3262473869},{"type":"v1","addr":"172.21.15.81:6821","nonce":3262473869}]},"public_addr":"172.21.15.81:6817/3262473869","cluster_addr":"172.21.15.81:6819/3262473869","heartbeat_back_addr":"172.21.15.81:6823/3262473869","heartbeat_front_addr":"172.21.15.81:6821/3262473869","state":["exists","up"]},{"osd":2,"uuid":"7da8ba88-6172-47a4-9284-457bd060469d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6809","nonce":1167487599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6811","nonce":1167487599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6815","nonce":1167487599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1167487599},{"type":"v1","addr":"172.21.15.81:6813","nonce":1167487599}]},"public_addr":"172.21.15.81:6809/1167487599","cluster_addr":"172.21.15.81:6811/1167487599","heartbeat_back_addr":"172.21.15.81:6815/1167487599","heartbeat_front_addr":"172.21.15.81:6813/1167487599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.415523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.487442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:05:06.468257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/539853599":"2023-10-14T01:44:41.496149+0000","172.21.15.81:0/2286238291":"2023-10-14T01:43:32.835955+0000","172.21.15.81:0/3013418471":"2023-10-14T01:41:04.918595+0000","172.21.15.81:0/2020033970":"2023-10-14T01:18:57.820662+0000","172.21.15.81:0/2598625570":"2023-10-14T01:17:38.993363+0000","172.21.15.81:0/2654634281":"2023-10-14T01:15:23.119762+0000","172.21.15.81:0/1063646072":"2023-10-14T01:31:44.757947+0000","172.21.15.81:0/2097712050":"2023-10-14T01:16:31.024332+0000","172.21.15.81:0/4121420252":"2023-10-14T01:34:12.634237+0000","172.21.15.81:0/1791769016":"2023-10-14T01:14:15.086879+0000","172.21.15.81:0/1915003886":"2023-10-14T01:24:41.876382+0000","172.21.15.81:0/1901974044":"2023-10-14T01:12:56.100890+0000","172.21.15.81:0/2292754641":"2023-10-14T01:42:24.957011+0000","172.21.15.81:0/1042564731":"2023-10-14T01:35:21.743201+0000","172.21.15.81:0/1782050421":"2023-10-14T01:21:15.020361+0000","172.21.15.81:0/805000911":"2023-10-14T01:30:35.970229+0000","172.21.15.81:0/3088214557":"2023-10-14T01:20:06.408921+0000","172.21.15.81:0/3155963726":"2023-10-14T01:37:39.646813+0000","172.21.15.81:0/2822357953":"2023-10-14T01:22:23.814651+0000","172.21.15.81:0/2894067023":"2023-10-14T01:36:30.532514+0000","172.21.15.81:0/3439709489":"2023-10-14T01:23:33.618654+0000","172.21.15.81:0/3249982660":"2023-10-14T01:25:50.920064+0000","172.21.15.81:0/2014696063":"2023-10-14T01:27:10.721467+0000","172.21.15.81:0/3583908646":"2023-10-14T01:29:27.587128+0000","172.21.15.81:0/3142402495":"2023-10-14T01:28:19.690907+0000","172.21.15.81:0/60711583":"2023-10-14T01:33:04.268367+0000","172.21.15.81:0/991631890":"2023-10-14T01:11:47.895058+0000","172.21.15.81:0/3240612452":"2023-10-14T01:38:48.108727+0000","172.21.15.81:0/880919163":"2023-10-14T01:39:56.845400+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T01:11:37.345 INFO:tasks.ceph:Scrubbing osd.0 2023-10-14T01:11:37.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:11:37.593 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2023-10-14T01:11:37.593 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:11:37.593 INFO:teuthology.orchestra.run.smithi081.stdout:} 2023-10-14T01:11:37.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-14T01:11:38.170 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 0 to deep-scrub 2023-10-14T01:11:38.195 INFO:tasks.ceph:Scrubbing osd.1 2023-10-14T01:11:38.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:11:38.459 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2023-10-14T01:11:38.460 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:11:38.460 INFO:teuthology.orchestra.run.smithi081.stdout:} 2023-10-14T01:11:38.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-14T01:11:39.097 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 1 to deep-scrub 2023-10-14T01:11:39.127 INFO:tasks.ceph:Scrubbing osd.2 2023-10-14T01:11:39.127 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:11:39.418 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2023-10-14T01:11:39.418 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:11:39.418 INFO:teuthology.orchestra.run.smithi081.stdout:} 2023-10-14T01:11:39.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-14T01:11:39.780 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 2 to deep-scrub 2023-10-14T01:11:39.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:11:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:11:40.335 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2111,"stamp":"2023-10-14T01:11:39.407671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26817810149,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1840775,"num_read_kb":7136910,"num_write":1804503,"num_write_kb":7217071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83700,"ondisk_log_size":83700,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17864124,"kb_used_data":14431716,"kb_used_omap":2016,"kb_used_meta":3430367,"kb_avail":265251396,"statfs":{"total":289910292480,"available":271617429504,"internally_reserved":0,"allocated":14778077184,"data_stored":23957870316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3512696363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,3,6,3,10,14,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16711680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4446,"num_read_kb":17724,"num_write":4293,"num_write_kb":17172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001849"},"pg_stats":[{"pgid":"2.7","version":"118'6018","reported_seq":8963,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.189850+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T01:11:37.189850+0000","last_peered":"2023-10-14T01:11:37.189850+0000","last_clean":"2023-10-14T01:11:37.189850+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:11:37.189850+0000","last_undegraded":"2023-10-14T01:11:37.189850+0000","last_fullsized":"2023-10-14T01:11:37.189850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:17:51.271717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'8964","reported_seq":14567,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.189969+0000","last_change":"2023-10-14T00:05:17.004867+0000","last_active":"2023-10-14T01:11:37.189969+0000","last_peered":"2023-10-14T01:11:37.189969+0000","last_clean":"2023-10-14T01:11:37.189969+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:11:37.189969+0000","last_undegraded":"2023-10-14T01:11:37.189969+0000","last_fullsized":"2023-10-14T01:11:37.189969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:48:27.138266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"118'5988","reported_seq":8733,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.184297+0000","last_change":"2023-10-14T00:05:17.002899+0000","last_active":"2023-10-14T01:11:37.184297+0000","last_peered":"2023-10-14T01:11:37.184297+0000","last_clean":"2023-10-14T01:11:37.184297+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:11:37.184297+0000","last_undegraded":"2023-10-14T01:11:37.184297+0000","last_fullsized":"2023-10-14T01:11:37.184297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:43:16.014296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"118'14776","reported_seq":22132,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.214438+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T01:11:37.214438+0000","last_peered":"2023-10-14T01:11:37.214438+0000","last_clean":"2023-10-14T01:11:37.214438+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:11:37.214438+0000","last_undegraded":"2023-10-14T01:11:37.214438+0000","last_fullsized":"2023-10-14T01:11:37.214438+0000","mapping_epoch":12,"log_start":"77'4700","ondisk_log_start":"77'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:37:32.329836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"118'467895","reported_seq":931596,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.383513+0000","last_change":"2023-10-14T00:20:08.229338+0000","last_active":"2023-10-14T01:11:37.383513+0000","last_peered":"2023-10-14T01:11:37.383513+0000","last_clean":"2023-10-14T01:11:37.383513+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:11:37.383513+0000","last_undegraded":"2023-10-14T01:11:37.383513+0000","last_fullsized":"2023-10-14T01:11:37.383513+0000","mapping_epoch":15,"log_start":"117'457833","ondisk_log_start":"117'457833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:31:10.509128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6926262272,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":466038,"num_read_kb":1833644,"num_write":465375,"num_write_kb":1861500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"118'3119","reported_seq":4667,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:35.560461+0000","last_change":"2023-10-14T00:05:17.000382+0000","last_active":"2023-10-14T01:11:35.560461+0000","last_peered":"2023-10-14T01:11:35.560461+0000","last_clean":"2023-10-14T01:11:35.560461+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:11:35.560461+0000","last_undegraded":"2023-10-14T01:11:35.560461+0000","last_fullsized":"2023-10-14T01:11:35.560461+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:02:11.052958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"118'444519","reported_seq":885259,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.209118+0000","last_change":"2023-10-14T00:20:08.199640+0000","last_active":"2023-10-14T01:11:37.209118+0000","last_peered":"2023-10-14T01:11:37.209118+0000","last_clean":"2023-10-14T01:11:37.209118+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:11:37.209118+0000","last_undegraded":"2023-10-14T01:11:37.209118+0000","last_fullsized":"2023-10-14T01:11:37.209118+0000","mapping_epoch":15,"log_start":"117'434423","ondisk_log_start":"117'434423","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:08:30.316718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6578409472,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":442952,"num_read_kb":1742264,"num_write":442123,"num_write_kb":1768492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"92'84","reported_seq":2778,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:33.615146+0000","last_change":"2023-10-14T00:05:16.999347+0000","last_active":"2023-10-14T01:11:33.615146+0000","last_peered":"2023-10-14T01:11:33.615146+0000","last_clean":"2023-10-14T01:11:33.615146+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:11:33.615146+0000","last_undegraded":"2023-10-14T01:11:33.615146+0000","last_fullsized":"2023-10-14T01:11:33.615146+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:57:01.029050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"118'462014","reported_seq":919957,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.385558+0000","last_change":"2023-10-14T00:20:08.294701+0000","last_active":"2023-10-14T01:11:37.385558+0000","last_peered":"2023-10-14T01:11:37.385558+0000","last_clean":"2023-10-14T01:11:37.385558+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:11:37.385558+0000","last_undegraded":"2023-10-14T01:11:37.385558+0000","last_fullsized":"2023-10-14T01:11:37.385558+0000","mapping_epoch":15,"log_start":"117'451927","ondisk_log_start":"117'451927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:26:25.273720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6824574976,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":460222,"num_read_kb":1810536,"num_write":459554,"num_write_kb":1838216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"93'119","reported_seq":2927,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:33.615332+0000","last_change":"2023-10-14T00:05:17.001059+0000","last_active":"2023-10-14T01:11:33.615332+0000","last_peered":"2023-10-14T01:11:33.615332+0000","last_clean":"2023-10-14T01:11:33.615332+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:11:33.615332+0000","last_undegraded":"2023-10-14T01:11:33.615332+0000","last_fullsized":"2023-10-14T01:11:33.615332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:16:12.406474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"118'439007","reported_seq":874651,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.217034+0000","last_change":"2023-10-14T00:20:08.207432+0000","last_active":"2023-10-14T01:11:37.217034+0000","last_peered":"2023-10-14T01:11:37.217034+0000","last_clean":"2023-10-14T01:11:37.217034+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:11:37.217034+0000","last_undegraded":"2023-10-14T01:11:37.217034+0000","last_fullsized":"2023-10-14T01:11:37.217034+0000","mapping_epoch":15,"log_start":"117'428922","ondisk_log_start":"117'428922","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:59:56.456702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6487969811,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":437815,"num_read_kb":1721660,"num_write":436655,"num_write_kb":1746614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'8905","reported_seq":12924,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.213437+0000","last_change":"2023-10-14T00:05:17.005268+0000","last_active":"2023-10-14T01:11:37.213437+0000","last_peered":"2023-10-14T01:11:37.213437+0000","last_clean":"2023-10-14T01:11:37.213437+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:11:37.213437+0000","last_undegraded":"2023-10-14T01:11:37.213437+0000","last_fullsized":"2023-10-14T01:11:37.213437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":8905,"log_dups_size":0,"ondisk_log_size":8905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:44:40.852686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":278,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:10:41.177657+0000","last_change":"2023-10-14T00:05:12.384839+0000","last_active":"2023-10-14T01:10:41.177657+0000","last_peered":"2023-10-14T01:10:41.177657+0000","last_clean":"2023-10-14T01:10:41.177657+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:10:41.177657+0000","last_undegraded":"2023-10-14T01:10:41.177657+0000","last_fullsized":"2023-10-14T01:10:41.177657+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:10.931569+0000","last_clean_scrub_stamp":"2023-10-14T00:05:10.931569+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:10:36.076607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26817216531,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1807027,"num_read_kb":7108104,"num_write":1803707,"num_write_kb":7214822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14775119872,"data_stored":23957483558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40330,"ondisk_log_size":40330,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43273,"ondisk_log_size":43273,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8152656,"kb_used_data":7213832,"kb_used_omap":1043,"kb_used_meta":937772,"kb_avail":86219184,"statfs":{"total":96636764160,"available":88288444416,"internally_reserved":0,"allocated":7386963968,"data_stored":11977377645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":960278775},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.17999999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.198,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.314},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":0.39900000000000002,"5min":27.538,"15min":27.538},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.17899999999999999,"15min":0.17499999999999999},"max":{"1min":3.298,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.42299999999999999,"15min":0.49099999999999999},"min":{"1min":0.24099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.339,"5min":3.339,"15min":12.361000000000001},"last":0.36299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739168,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4602124,"kb_used_data":3516548,"kb_used_omap":472,"kb_used_meta":1085095,"kb_avail":89769716,"statfs":{"total":96636764160,"available":91924189184,"internally_reserved":0,"allocated":3600945152,"data_stored":5841433585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1111137999},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.39600000000000002,"15min":0.42899999999999999},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.16400000000000001},"max":{"1min":0.58899999999999997,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.60299999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":1.837,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.497,"15min":0.51400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":1.881,"5min":15.891,"15min":15.891},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.47099999999999997,"15min":0.57399999999999995},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":0.56299999999999994,"5min":5.6699999999999999,"15min":9.673},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359739167,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5109344,"kb_used_data":3701336,"kb_used_omap":500,"kb_used_meta":1407499,"kb_avail":89262496,"statfs":{"total":96636764160,"available":91404795904,"internally_reserved":0,"allocated":3790168064,"data_stored":6139059086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1441279589},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,0,5,8,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.45700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":5.7729999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":3.5049999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.47699999999999998,"15min":0.53400000000000003},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":5.1909999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":3.1629999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.53900000000000003},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":18.966999999999999},"last":3.532}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3788517376,"data_stored":6138245120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3599470592,"data_stored":5840924691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7387131904,"data_stored":11978313747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:40.338 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.339 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.339 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.340 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.340 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.340 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.341 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.341 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.342 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.342 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.342 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.343 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.343 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-14T00:05:10.931569+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=10, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:40.343 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:46.777 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:11:46.776+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:12:00.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:00.843 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:12:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:12:00.880 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2122,"stamp":"2023-10-14T01:11:59.417999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26849205989,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1848860,"num_read_kb":7169090,"num_write":1812518,"num_write_kb":7249131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11892,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83712,"ondisk_log_size":83712,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17843768,"kb_used_data":14495584,"kb_used_omap":2016,"kb_used_meta":3346143,"kb_avail":265271752,"statfs":{"total":289910292480,"available":271638274048,"internally_reserved":0,"allocated":14843478016,"data_stored":24023248495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3426450987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,8,7,13,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15773696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4058,"num_read_kb":16152,"num_write":4007,"num_write_kb":16028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008564"},"pg_stats":[{"pgid":"2.7","version":"119'6046","reported_seq":9001,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.258177+0000","last_change":"2023-10-14T00:05:17.002881+0000","last_active":"2023-10-14T01:11:57.258177+0000","last_peered":"2023-10-14T01:11:57.258177+0000","last_clean":"2023-10-14T01:11:57.258177+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:11:57.258177+0000","last_undegraded":"2023-10-14T01:11:57.258177+0000","last_fullsized":"2023-10-14T01:11:57.258177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":6046,"log_dups_size":0,"ondisk_log_size":6046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'9016","reported_seq":14641,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.271831+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:11:57.271831+0000","last_peered":"2023-10-14T01:11:57.271831+0000","last_clean":"2023-10-14T01:11:57.271831+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:11:57.271831+0000","last_undegraded":"2023-10-14T01:11:57.271831+0000","last_fullsized":"2023-10-14T01:11:57.271831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'6030","reported_seq":8793,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.241380+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:11:57.241380+0000","last_peered":"2023-10-14T01:11:57.241380+0000","last_clean":"2023-10-14T01:11:57.241380+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:11:57.241380+0000","last_undegraded":"2023-10-14T01:11:57.241380+0000","last_fullsized":"2023-10-14T01:11:57.241380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6030,"log_dups_size":0,"ondisk_log_size":6030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"119'14846","reported_seq":22224,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.283503+0000","last_change":"2023-10-14T00:05:17.005947+0000","last_active":"2023-10-14T01:11:57.283503+0000","last_peered":"2023-10-14T01:11:57.283503+0000","last_clean":"2023-10-14T01:11:57.283503+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:11:57.283503+0000","last_undegraded":"2023-10-14T01:11:57.283503+0000","last_fullsized":"2023-10-14T01:11:57.283503+0000","mapping_epoch":12,"log_start":"77'4800","ondisk_log_start":"77'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:15.963512+0000","last_clean_scrub_stamp":"2023-10-14T00:05:15.963512+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"119'470017","reported_seq":935768,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:57.498005+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:11:57.498005+0000","last_peered":"2023-10-14T01:11:57.498005+0000","last_clean":"2023-10-14T01:11:57.498005+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:11:57.498005+0000","last_undegraded":"2023-10-14T01:11:57.498005+0000","last_fullsized":"2023-10-14T01:11:57.498005+0000","mapping_epoch":15,"log_start":"117'459933","ondisk_log_start":"117'459933","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":77,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6934396928,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":468078,"num_read_kb":1841748,"num_write":467442,"num_write_kb":1869768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"119'3141","reported_seq":4703,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:55.373571+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:11:55.373571+0000","last_peered":"2023-10-14T01:11:55.373571+0000","last_clean":"2023-10-14T01:11:55.373571+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:11:55.373571+0000","last_undegraded":"2023-10-14T01:11:55.373571+0000","last_fullsized":"2023-10-14T01:11:55.373571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3141,"log_dups_size":0,"ondisk_log_size":3141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"119'446450","reported_seq":889179,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.292755+0000","last_change":"2023-10-14T00:20:08.199640+0000","last_active":"2023-10-14T01:11:57.292755+0000","last_peered":"2023-10-14T01:11:57.292755+0000","last_clean":"2023-10-14T01:11:57.292755+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:11:57.292755+0000","last_undegraded":"2023-10-14T01:11:57.292755+0000","last_fullsized":"2023-10-14T01:11:57.292755+0000","mapping_epoch":15,"log_start":"117'436423","ondisk_log_start":"117'436423","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6586019840,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":444939,"num_read_kb":1750176,"num_write":444054,"num_write_kb":1776216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2807,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:54.096925+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:11:54.096925+0000","last_peered":"2023-10-14T01:11:54.096925+0000","last_clean":"2023-10-14T01:11:54.096925+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:11:54.096925+0000","last_undegraded":"2023-10-14T01:11:54.096925+0000","last_fullsized":"2023-10-14T01:11:54.096925+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"119'464140","reported_seq":924152,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:57.459139+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:11:57.459139+0000","last_peered":"2023-10-14T01:11:57.459139+0000","last_clean":"2023-10-14T01:11:57.459139+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:11:57.459139+0000","last_undegraded":"2023-10-14T01:11:57.459139+0000","last_fullsized":"2023-10-14T01:11:57.459139+0000","mapping_epoch":15,"log_start":"117'454127","ondisk_log_start":"117'454127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":65,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6832685056,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":462286,"num_read_kb":1818764,"num_write":461634,"num_write_kb":1846536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":2957,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:54.097737+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:11:54.097737+0000","last_peered":"2023-10-14T01:11:54.097737+0000","last_clean":"2023-10-14T01:11:54.097737+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:11:54.097737+0000","last_undegraded":"2023-10-14T01:11:54.097737+0000","last_fullsized":"2023-10-14T01:11:54.097737+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"119'440944","reported_seq":878586,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.277364+0000","last_change":"2023-10-14T00:20:08.207432+0000","last_active":"2023-10-14T01:11:57.277364+0000","last_peered":"2023-10-14T01:11:57.277364+0000","last_clean":"2023-10-14T01:11:57.277364+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:11:57.277364+0000","last_undegraded":"2023-10-14T01:11:57.277364+0000","last_fullsized":"2023-10-14T01:11:57.277364+0000","mapping_epoch":15,"log_start":"117'430922","ondisk_log_start":"117'430922","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6495510547,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":439809,"num_read_kb":1729596,"num_write":438592,"num_write_kb":1754362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'8964","reported_seq":13005,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.276170+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:11:57.276170+0000","last_peered":"2023-10-14T01:11:57.276170+0000","last_clean":"2023-10-14T01:11:57.276170+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:11:57.276170+0000","last_undegraded":"2023-10-14T01:11:57.276170+0000","last_fullsized":"2023-10-14T01:11:57.276170+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":290,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:43.787781+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:11:43.787781+0000","last_peered":"2023-10-14T01:11:43.787781+0000","last_clean":"2023-10-14T01:11:43.787781+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:11:43.787781+0000","last_undegraded":"2023-10-14T01:11:43.787781+0000","last_fullsized":"2023-10-14T01:11:43.787781+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26848612371,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1815112,"num_read_kb":7140284,"num_write":1811722,"num_write_kb":7246882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14840840192,"data_stored":24023203878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40146,"ondisk_log_size":40146,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11892,"num_omap_keys":217,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43467,"ondisk_log_size":43467,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739173,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8042140,"kb_used_data":7245844,"kb_used_omap":1043,"kb_used_meta":795244,"kb_avail":86329700,"statfs":{"total":96636764160,"available":88401612800,"internally_reserved":0,"allocated":7419744256,"data_stored":12010150382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":814330103},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.17999999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.198,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.247},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":0.39900000000000002,"5min":27.538,"15min":27.538},"last":0.31900000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.495,"5min":3.298,"15min":4.8739999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.42899999999999999,"15min":0.499},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":2.0750000000000002,"5min":3.339,"15min":12.361000000000001},"last":0.33300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739172,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4635532,"kb_used_data":3531908,"kb_used_omap":472,"kb_used_meta":1103143,"kb_avail":89736308,"statfs":{"total":96636764160,"available":91889979392,"internally_reserved":0,"allocated":3616673792,"data_stored":5857154674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1129619151},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.39600000000000002,"15min":0.42899999999999999},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.16400000000000001},"max":{"1min":0.58899999999999997,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.60299999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":1.837,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.247}]},{"osd":2,"last update":"Sat Oct 14 01:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.497,"15min":0.51400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":1.881,"5min":15.891,"15min":15.891},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.47099999999999997,"15min":0.57399999999999995},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":0.56299999999999994,"5min":5.6699999999999999,"15min":9.673},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739171,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5166096,"kb_used_data":3717832,"kb_used_omap":500,"kb_used_meta":1447755,"kb_avail":89205744,"statfs":{"total":96636764160,"available":91346681856,"internally_reserved":0,"allocated":3807059968,"data_stored":6155943439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1482501733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,3,10,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.45700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":3.5390000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.47699999999999998,"15min":0.53400000000000003},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":5.1909999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.221},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.53900000000000003},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3805499392,"data_stored":6155227136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3615318016,"data_stored":5856772115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7420022784,"data_stored":12011204627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:12:00.882 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:00.882 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-14T00:05:15.963512+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=15, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:00.883 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:00.883 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:00.883 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:00.884 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:00.884 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:20.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:21.369 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:12:21.370 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:12:21.406 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2132,"stamp":"2023-10-14T01:12:19.421451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26879205093,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1856566,"num_read_kb":7199770,"num_write":1820194,"num_write_kb":7279835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83952,"ondisk_log_size":83952,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18152248,"kb_used_data":14557088,"kb_used_omap":2016,"kb_used_meta":3593119,"kb_avail":264963272,"statfs":{"total":289910292480,"available":271322390528,"internally_reserved":0,"allocated":14906458112,"data_stored":24086228591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3679354411},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,0,13,10,16,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18550784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4792,"num_read_kb":19108,"num_write":4753,"num_write_kb":19012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002098"},"pg_stats":[{"pgid":"2.7","version":"119'6073","reported_seq":9044,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.575422+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:12:14.575422+0000","last_peered":"2023-10-14T01:12:14.575422+0000","last_clean":"2023-10-14T01:12:14.575422+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:12:14.575422+0000","last_undegraded":"2023-10-14T01:12:14.575422+0000","last_fullsized":"2023-10-14T01:12:14.575422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6073,"log_dups_size":0,"ondisk_log_size":6073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'9037","reported_seq":14674,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.575225+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:12:14.575225+0000","last_peered":"2023-10-14T01:12:14.575225+0000","last_clean":"2023-10-14T01:12:14.575225+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:12:14.575225+0000","last_undegraded":"2023-10-14T01:12:14.575225+0000","last_fullsized":"2023-10-14T01:12:14.575225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9037,"log_dups_size":0,"ondisk_log_size":9037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'6044","reported_seq":8815,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.575007+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:12:14.575007+0000","last_peered":"2023-10-14T01:12:14.575007+0000","last_clean":"2023-10-14T01:12:14.575007+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:12:14.575007+0000","last_undegraded":"2023-10-14T01:12:14.575007+0000","last_fullsized":"2023-10-14T01:12:14.575007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6044,"log_dups_size":0,"ondisk_log_size":6044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"119'14896","reported_seq":22302,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.575842+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:12:14.575842+0000","last_peered":"2023-10-14T01:12:14.575842+0000","last_clean":"2023-10-14T01:12:14.575842+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:12:14.575842+0000","last_undegraded":"2023-10-14T01:12:14.575842+0000","last_fullsized":"2023-10-14T01:12:14.575842+0000","mapping_epoch":12,"log_start":"77'4800","ondisk_log_start":"77'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"119'472106","reported_seq":939871,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:17.680221+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:12:17.680221+0000","last_peered":"2023-10-14T01:12:17.680221+0000","last_clean":"2023-10-14T01:12:17.680221+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:12:17.680221+0000","last_undegraded":"2023-10-14T01:12:17.680221+0000","last_fullsized":"2023-10-14T01:12:17.680221+0000","mapping_epoch":15,"log_start":"118'462033","ondisk_log_start":"118'462033","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":161,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 32s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6942253056,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":470079,"num_read_kb":1849716,"num_write":469474,"num_write_kb":1877896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"119'3154","reported_seq":4720,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:16.487835+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:12:16.487835+0000","last_peered":"2023-10-14T01:12:16.487835+0000","last_clean":"2023-10-14T01:12:16.487835+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:12:16.487835+0000","last_undegraded":"2023-10-14T01:12:16.487835+0000","last_fullsized":"2023-10-14T01:12:16.487835+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3154,"log_dups_size":0,"ondisk_log_size":3154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"119'448279","reported_seq":892878,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:17.360253+0000","last_change":"2023-10-14T01:12:10.485851+0000","last_active":"2023-10-14T01:12:17.360253+0000","last_peered":"2023-10-14T01:12:17.360253+0000","last_clean":"2023-10-14T01:12:17.360253+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:12:17.360253+0000","last_undegraded":"2023-10-14T01:12:17.360253+0000","last_fullsized":"2023-10-14T01:12:17.360253+0000","mapping_epoch":15,"log_start":"117'438223","ondisk_log_start":"117'438223","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6593277952,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":446809,"num_read_kb":1757624,"num_write":445883,"num_write_kb":1783532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2815,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.579657+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:12:14.579657+0000","last_peered":"2023-10-14T01:12:14.579657+0000","last_clean":"2023-10-14T01:12:14.579657+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:12:14.579657+0000","last_undegraded":"2023-10-14T01:12:14.579657+0000","last_fullsized":"2023-10-14T01:12:14.579657+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"119'466154","reported_seq":928166,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:17.643685+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:12:17.643685+0000","last_peered":"2023-10-14T01:12:17.643685+0000","last_clean":"2023-10-14T01:12:17.643685+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:12:17.643685+0000","last_undegraded":"2023-10-14T01:12:17.643685+0000","last_fullsized":"2023-10-14T01:12:17.643685+0000","mapping_epoch":15,"log_start":"118'456128","ondisk_log_start":"118'456128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":203,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6840184832,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":464270,"num_read_kb":1826664,"num_write":463555,"num_write_kb":1854220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":2965,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.575190+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:12:14.575190+0000","last_peered":"2023-10-14T01:12:14.575190+0000","last_clean":"2023-10-14T01:12:14.575190+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:12:14.575190+0000","last_undegraded":"2023-10-14T01:12:14.575190+0000","last_fullsized":"2023-10-14T01:12:14.575190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"119'442907","reported_seq":882407,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:17.365282+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:12:17.365282+0000","last_peered":"2023-10-14T01:12:17.365282+0000","last_clean":"2023-10-14T01:12:17.365282+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:12:17.365282+0000","last_undegraded":"2023-10-14T01:12:17.365282+0000","last_fullsized":"2023-10-14T01:12:17.365282+0000","mapping_epoch":15,"log_start":"117'432822","ondisk_log_start":"117'432822","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":101,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6502895635,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":441660,"num_read_kb":1736960,"num_write":440486,"num_write_kb":1761938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'8985","reported_seq":13038,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:14.576221+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:12:14.576221+0000","last_peered":"2023-10-14T01:12:14.576221+0000","last_clean":"2023-10-14T01:12:14.576221+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:12:14.576221+0000","last_undegraded":"2023-10-14T01:12:14.576221+0000","last_fullsized":"2023-10-14T01:12:14.576221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":290,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:43.787781+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:11:43.787781+0000","last_peered":"2023-10-14T01:11:43.787781+0000","last_clean":"2023-10-14T01:11:43.787781+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:11:43.787781+0000","last_undegraded":"2023-10-14T01:11:43.787781+0000","last_fullsized":"2023-10-14T01:11:43.787781+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26878611475,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1822818,"num_read_kb":7170964,"num_write":1819398,"num_write_kb":7277586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14903689216,"data_stored":24086052902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43613,"ondisk_log_size":43613,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739177,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8197888,"kb_used_data":7276728,"kb_used_omap":1043,"kb_used_meta":920108,"kb_avail":86173952,"statfs":{"total":96636764160,"available":88242126848,"internally_reserved":0,"allocated":7451369472,"data_stored":12041775598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":942190839},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.17999999999999999,"5min":0.159,"15min":0.159},"max":{"1min":2.198,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.255},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.64500000000000002,"15min":0.56699999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":0.39900000000000002,"5min":27.538,"15min":27.538},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.495,"5min":3.298,"15min":4.8739999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.42899999999999999,"15min":0.499},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":2.0750000000000002,"5min":3.339,"15min":12.361000000000001},"last":0.45200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739176,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4718436,"kb_used_data":3546844,"kb_used_omap":472,"kb_used_meta":1171111,"kb_avail":89653404,"statfs":{"total":96636764160,"available":91805085696,"internally_reserved":0,"allocated":3631968256,"data_stored":5872449138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1199218383},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,3,5,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.39600000000000002,"15min":0.42899999999999999},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.16400000000000001},"max":{"1min":0.58899999999999997,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.60299999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":1.837,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.497,"15min":0.51400000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":1.881,"5min":15.891,"15min":15.891},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.47099999999999997,"15min":0.57399999999999995},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":0.56299999999999994,"5min":5.6699999999999999,"15min":9.673},"last":0.42299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739175,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5235924,"kb_used_data":3733516,"kb_used_omap":500,"kb_used_meta":1501899,"kb_avail":89135916,"statfs":{"total":96636764160,"available":91275177984,"internally_reserved":0,"allocated":3823120384,"data_stored":6172003855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1537945189},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,8,4,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.45700000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.47699999999999998,"15min":0.53400000000000003},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":5.1909999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.255},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.53900000000000003},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.215}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3821686784,"data_stored":6171414528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3630563328,"data_stored":5872017427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7451439104,"data_stored":12042620947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:12:21.408 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:21.408 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:21.409 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:21.409 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:21.410 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:30.259 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:12:30.251+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:12:30.264 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:12:30.259+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:12:41.411 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:41.927 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:12:41.928 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:12:41.975 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2143,"stamp":"2023-10-14T01:12:41.425194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26908983013,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1864338,"num_read_kb":7230730,"num_write":1827806,"num_write_kb":7310283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83983,"ondisk_log_size":83983,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18218728,"kb_used_data":14618128,"kb_used_omap":2016,"kb_used_meta":3598559,"kb_avail":264896792,"statfs":{"total":289910292480,"available":271254315008,"internally_reserved":0,"allocated":14968963072,"data_stored":24148733551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3684924971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,7,10,8,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14815232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3859,"num_read_kb":15372,"num_write":3800,"num_write_kb":15200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002016"},"pg_stats":[{"pgid":"2.7","version":"119'6093","reported_seq":9072,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:36.747264+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:12:36.747264+0000","last_peered":"2023-10-14T01:12:36.747264+0000","last_clean":"2023-10-14T01:12:36.747264+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:12:36.747264+0000","last_undegraded":"2023-10-14T01:12:36.747264+0000","last_fullsized":"2023-10-14T01:12:36.747264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'9067","reported_seq":14717,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:36.750239+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:12:36.750239+0000","last_peered":"2023-10-14T01:12:36.750239+0000","last_clean":"2023-10-14T01:12:36.750239+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:12:36.750239+0000","last_undegraded":"2023-10-14T01:12:36.750239+0000","last_fullsized":"2023-10-14T01:12:36.750239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'6065","reported_seq":8845,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:36.738175+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:12:36.738175+0000","last_peered":"2023-10-14T01:12:36.738175+0000","last_clean":"2023-10-14T01:12:36.738175+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:12:36.738175+0000","last_undegraded":"2023-10-14T01:12:36.738175+0000","last_fullsized":"2023-10-14T01:12:36.738175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"119'14946","reported_seq":22373,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:36.765310+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:12:36.765310+0000","last_peered":"2023-10-14T01:12:36.765310+0000","last_clean":"2023-10-14T01:12:36.765310+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:12:36.765310+0000","last_undegraded":"2023-10-14T01:12:36.765310+0000","last_fullsized":"2023-10-14T01:12:36.765310+0000","mapping_epoch":12,"log_start":"79'4900","ondisk_log_start":"79'4900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"119'474112","reported_seq":943859,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:37.747043+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:12:37.747043+0000","last_peered":"2023-10-14T01:12:37.747043+0000","last_clean":"2023-10-14T01:12:37.747043+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:12:37.747043+0000","last_undegraded":"2023-10-14T01:12:37.747043+0000","last_fullsized":"2023-10-14T01:12:37.747043+0000","mapping_epoch":15,"log_start":"118'464033","ondisk_log_start":"118'464033","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":294,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 52s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6949777408,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":472053,"num_read_kb":1857568,"num_write":471389,"num_write_kb":1885556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"119'3167","reported_seq":4738,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.070207+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:12:35.070207+0000","last_peered":"2023-10-14T01:12:35.070207+0000","last_clean":"2023-10-14T01:12:35.070207+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:12:35.070207+0000","last_undegraded":"2023-10-14T01:12:35.070207+0000","last_fullsized":"2023-10-14T01:12:35.070207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"119'450138","reported_seq":896662,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:37.431826+0000","last_change":"2023-10-14T01:12:26.729238+0000","last_active":"2023-10-14T01:12:37.431826+0000","last_peered":"2023-10-14T01:12:37.431826+0000","last_clean":"2023-10-14T01:12:37.431826+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:12:37.431826+0000","last_undegraded":"2023-10-14T01:12:37.431826+0000","last_fullsized":"2023-10-14T01:12:37.431826+0000","mapping_epoch":15,"log_start":"118'440123","ondisk_log_start":"118'440123","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6600503296,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":448730,"num_read_kb":1765272,"num_write":447742,"num_write_kb":1790968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2823,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.056700+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:12:35.056700+0000","last_peered":"2023-10-14T01:12:35.056700+0000","last_clean":"2023-10-14T01:12:35.056700+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:12:35.056700+0000","last_undegraded":"2023-10-14T01:12:35.056700+0000","last_fullsized":"2023-10-14T01:12:35.056700+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"119'468206","reported_seq":932280,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:37.744739+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:12:37.744739+0000","last_peered":"2023-10-14T01:12:37.744739+0000","last_clean":"2023-10-14T01:12:37.744739+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:12:37.744739+0000","last_undegraded":"2023-10-14T01:12:37.744739+0000","last_fullsized":"2023-10-14T01:12:37.744739+0000","mapping_epoch":15,"log_start":"118'458128","ondisk_log_start":"118'458128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":327,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 53s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6847873024,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":466316,"num_read_kb":1834820,"num_write":465523,"num_write_kb":1862092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":2973,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.055740+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:12:35.055740+0000","last_peered":"2023-10-14T01:12:35.055740+0000","last_clean":"2023-10-14T01:12:35.055740+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:12:35.055740+0000","last_undegraded":"2023-10-14T01:12:35.055740+0000","last_fullsized":"2023-10-14T01:12:35.055740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"119'444856","reported_seq":886198,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:37.430758+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:12:37.430758+0000","last_peered":"2023-10-14T01:12:37.430758+0000","last_clean":"2023-10-14T01:12:37.430758+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:12:37.430758+0000","last_undegraded":"2023-10-14T01:12:37.430758+0000","last_fullsized":"2023-10-14T01:12:37.430758+0000","mapping_epoch":15,"log_start":"118'434822","ondisk_log_start":"118'434822","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":219,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 31s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6510235667,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":443491,"num_read_kb":1744264,"num_write":442356,"num_write_kb":1769418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'9016","reported_seq":13082,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:36.803537+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:12:36.803537+0000","last_peered":"2023-10-14T01:12:36.803537+0000","last_clean":"2023-10-14T01:12:36.803537+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:12:36.803537+0000","last_undegraded":"2023-10-14T01:12:36.803537+0000","last_fullsized":"2023-10-14T01:12:36.803537+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":290,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-14T01:11:43.787781+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:11:43.787781+0000","last_peered":"2023-10-14T01:11:43.787781+0000","last_clean":"2023-10-14T01:11:43.787781+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:11:43.787781+0000","last_undegraded":"2023-10-14T01:11:43.787781+0000","last_fullsized":"2023-10-14T01:11:43.787781+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26908389395,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1830590,"num_read_kb":7201924,"num_write":1827010,"num_write_kb":7308034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14965964800,"data_stored":24148328486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40206,"ondisk_log_size":40206,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43678,"ondisk_log_size":43678,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739181,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8179720,"kb_used_data":7307264,"kb_used_omap":1043,"kb_used_meta":871404,"kb_avail":86192120,"statfs":{"total":96636764160,"available":88260730880,"internally_reserved":0,"allocated":7482638336,"data_stored":12073044462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":892317943},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.56299999999999994,"15min":0.46800000000000003},"min":{"1min":0.19800000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.47799999999999998,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.52500000000000002,"15min":0.56399999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.183},"max":{"1min":0.40300000000000002,"5min":27.538,"15min":27.538},"last":0.255}]},{"osd":1,"last update":"Sat Oct 14 01:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.40500000000000003,"15min":0.40400000000000003},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.495,"5min":3.298,"15min":4.8739999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.42899999999999999,"15min":0.499},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":2.0750000000000002,"5min":3.339,"15min":12.361000000000001},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739180,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4751304,"kb_used_data":3561664,"kb_used_omap":472,"kb_used_meta":1189159,"kb_avail":89620536,"statfs":{"total":96636764160,"available":91771428864,"internally_reserved":0,"allocated":3647143936,"data_stored":5887624818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1217699535},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,4,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.37,"15min":0.442},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16400000000000001},"max":{"1min":4.0579999999999998,"5min":4.0579999999999998,"15min":5.3300000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.53900000000000003},"min":{"1min":0.22,"5min":0.20599999999999999,"15min":0.20000000000000001},"max":{"1min":4.2329999999999997,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.311}]},{"osd":2,"last update":"Sat Oct 14 01:12:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.58799999999999997,"15min":0.54300000000000004},"min":{"1min":0.20100000000000001,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":4.1289999999999996,"5min":15.891,"15min":15.891},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.43099999999999999,"15min":0.58599999999999997},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":9.673},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739179,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5287704,"kb_used_data":3749200,"kb_used_omap":500,"kb_used_meta":1537995,"kb_avail":89084136,"statfs":{"total":96636764160,"available":91222155264,"internally_reserved":0,"allocated":3839180800,"data_stored":6188064271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1574907493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53900000000000003,"15min":0.47499999999999998},"min":{"1min":0.188,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.50800000000000001,"15min":0.46000000000000002},"min":{"1min":0.184,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.5390000000000001,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.52300000000000002},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":3.1629999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.53700000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.532,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3837591552,"data_stored":6187319296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3645841408,"data_stored":5887295507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7482531840,"data_stored":12073713683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:12:41.977 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:41.977 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:41.977 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:41.978 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:41.978 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:01.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:02.527 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:13:02.528 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:13:02.567 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2154,"stamp":"2023-10-14T01:13:01.428661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26938564325,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1872051,"num_read_kb":7261502,"num_write":1835407,"num_write_kb":7340687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84035,"ondisk_log_size":84035,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18477416,"kb_used_data":14678924,"kb_used_omap":2016,"kb_used_meta":3796447,"kb_avail":264638104,"statfs":{"total":289910292480,"available":270989418496,"internally_reserved":0,"allocated":15031218176,"data_stored":24210965873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":3887562283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,13,19,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14897152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3882,"num_read_kb":15484,"num_write":3820,"num_write_kb":15280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001769"},"pg_stats":[{"pgid":"2.7","version":"119'6123","reported_seq":9112,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.536565+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:12:55.536565+0000","last_peered":"2023-10-14T01:12:55.536565+0000","last_clean":"2023-10-14T01:12:55.536565+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:12:55.536565+0000","last_undegraded":"2023-10-14T01:12:55.536565+0000","last_fullsized":"2023-10-14T01:12:55.536565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'9112","reported_seq":14775,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.536482+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:12:55.536482+0000","last_peered":"2023-10-14T01:12:55.536482+0000","last_clean":"2023-10-14T01:12:55.536482+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:12:55.536482+0000","last_undegraded":"2023-10-14T01:12:55.536482+0000","last_fullsized":"2023-10-14T01:12:55.536482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'6095","reported_seq":8884,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.535584+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:12:55.535584+0000","last_peered":"2023-10-14T01:12:55.535584+0000","last_clean":"2023-10-14T01:12:55.535584+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:12:55.535584+0000","last_undegraded":"2023-10-14T01:12:55.535584+0000","last_fullsized":"2023-10-14T01:12:55.535584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"119'15021","reported_seq":22469,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.536402+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:12:55.536402+0000","last_peered":"2023-10-14T01:12:55.536402+0000","last_clean":"2023-10-14T01:12:55.536402+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:12:55.536402+0000","last_undegraded":"2023-10-14T01:12:55.536402+0000","last_fullsized":"2023-10-14T01:12:55.536402+0000","mapping_epoch":12,"log_start":"79'5000","ondisk_log_start":"79'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"120'476177","reported_seq":947971,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:57.873248+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:12:57.873248+0000","last_peered":"2023-10-14T01:12:57.873248+0000","last_clean":"2023-10-14T01:12:57.873248+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:12:57.873248+0000","last_undegraded":"2023-10-14T01:12:57.873248+0000","last_fullsized":"2023-10-14T01:12:57.873248+0000","mapping_epoch":15,"log_start":"118'466133","ondisk_log_start":"118'466133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":365,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 72s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6957637632,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":474086,"num_read_kb":1865680,"num_write":473403,"num_write_kb":1893612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"119'3173","reported_seq":4749,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.535159+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:12:55.535159+0000","last_peered":"2023-10-14T01:12:55.535159+0000","last_clean":"2023-10-14T01:12:55.535159+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:12:55.535159+0000","last_undegraded":"2023-10-14T01:12:55.535159+0000","last_fullsized":"2023-10-14T01:12:55.535159+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"120'452049","reported_seq":900482,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:12:57.523153+0000","last_change":"2023-10-14T01:12:52.867888+0000","last_active":"2023-10-14T01:12:57.523153+0000","last_peered":"2023-10-14T01:12:57.523153+0000","last_clean":"2023-10-14T01:12:57.523153+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:12:57.523153+0000","last_undegraded":"2023-10-14T01:12:57.523153+0000","last_fullsized":"2023-10-14T01:12:57.523153+0000","mapping_epoch":15,"log_start":"118'442023","ondisk_log_start":"118'442023","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6607953920,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":450629,"num_read_kb":1772848,"num_write":449653,"num_write_kb":1798612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2832,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.535108+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:12:55.535108+0000","last_peered":"2023-10-14T01:12:55.535108+0000","last_clean":"2023-10-14T01:12:55.535108+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:12:55.535108+0000","last_undegraded":"2023-10-14T01:12:55.535108+0000","last_fullsized":"2023-10-14T01:12:55.535108+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"120'470172","reported_seq":936188,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:57.869783+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:12:57.869783+0000","last_peered":"2023-10-14T01:12:57.869783+0000","last_clean":"2023-10-14T01:12:57.869783+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:12:57.869783+0000","last_undegraded":"2023-10-14T01:12:57.869783+0000","last_fullsized":"2023-10-14T01:12:57.869783+0000","mapping_epoch":15,"log_start":"118'460128","ondisk_log_start":"118'460128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":429,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 73s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6855184384,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":468246,"num_read_kb":1842524,"num_write":467420,"num_write_kb":1869680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":2982,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.546785+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:12:55.546785+0000","last_peered":"2023-10-14T01:12:55.546785+0000","last_clean":"2023-10-14T01:12:55.546785+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:12:55.546785+0000","last_undegraded":"2023-10-14T01:12:55.546785+0000","last_fullsized":"2023-10-14T01:12:55.546785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"120'446735","reported_seq":889948,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:57.525686+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:12:57.525686+0000","last_peered":"2023-10-14T01:12:57.525686+0000","last_clean":"2023-10-14T01:12:57.525686+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:12:57.525686+0000","last_undegraded":"2023-10-14T01:12:57.525686+0000","last_fullsized":"2023-10-14T01:12:57.525686+0000","mapping_epoch":15,"log_start":"118'436722","ondisk_log_start":"118'436722","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":367,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 51s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6517194771,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":445342,"num_read_kb":1751644,"num_write":444135,"num_write_kb":1776534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'9061","reported_seq":13140,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:12:55.536922+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:12:55.536922+0000","last_peered":"2023-10-14T01:12:55.536922+0000","last_clean":"2023-10-14T01:12:55.536922+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:12:55.536922+0000","last_undegraded":"2023-10-14T01:12:55.536922+0000","last_fullsized":"2023-10-14T01:12:55.536922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9061,"log_dups_size":0,"ondisk_log_size":9061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":291,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:51.191389+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:11:51.191389+0000","last_peered":"2023-10-14T01:11:51.191389+0000","last_clean":"2023-10-14T01:11:51.191389+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:11:51.191389+0000","last_undegraded":"2023-10-14T01:11:51.191389+0000","last_fullsized":"2023-10-14T01:11:51.191389+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26937970707,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1838303,"num_read_kb":7232696,"num_write":1834611,"num_write_kb":7338438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15028137984,"data_stored":24210501670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40127,"ondisk_log_size":40127,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":43809,"ondisk_log_size":43809,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739185,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8295988,"kb_used_data":7337516,"kb_used_omap":1043,"kb_used_meta":957420,"kb_avail":86075852,"statfs":{"total":96636764160,"available":88141672448,"internally_reserved":0,"allocated":7513616384,"data_stored":12104014916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":980398327},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.56299999999999994,"15min":0.46800000000000003},"min":{"1min":0.19800000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.47799999999999998,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.52500000000000002,"15min":0.56399999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.183},"max":{"1min":0.40300000000000002,"5min":27.538,"15min":27.538},"last":0.41499999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:12:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36799999999999999,"15min":0.40300000000000002},"min":{"1min":0.214,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.627,"5min":3.298,"15min":4.8739999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41399999999999998,"15min":0.45400000000000001},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.60699999999999998,"5min":3.339,"15min":12.361000000000001},"last":0.29699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739184,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4823952,"kb_used_data":3576580,"kb_used_omap":472,"kb_used_meta":1246887,"kb_avail":89547888,"statfs":{"total":96636764160,"available":91697037312,"internally_reserved":0,"allocated":3662417920,"data_stored":5902891208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1276813007},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.37,"15min":0.442},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16400000000000001},"max":{"1min":4.0579999999999998,"5min":4.0579999999999998,"15min":5.3300000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.53900000000000003},"min":{"1min":0.22,"5min":0.20599999999999999,"15min":0.20000000000000001},"max":{"1min":4.2329999999999997,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:12:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.58799999999999997,"15min":0.54300000000000004},"min":{"1min":0.20100000000000001,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":4.1289999999999996,"5min":15.891,"15min":15.891},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.43099999999999999,"15min":0.58599999999999997},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":9.673},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739183,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5357476,"kb_used_data":3764828,"kb_used_omap":500,"kb_used_meta":1592139,"kb_avail":89014364,"statfs":{"total":96636764160,"available":91150708736,"internally_reserved":0,"allocated":3855183872,"data_stored":6204059749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1630350949},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53900000000000003,"15min":0.47499999999999998},"min":{"1min":0.188,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.245},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.50800000000000001,"15min":0.46000000000000002},"min":{"1min":0.184,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.5390000000000001,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":3.0830000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.52300000000000002},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":3.1629999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.53700000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.532,"5min":4.9690000000000003,"15min":18.966999999999999},"last":3.125}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3853611008,"data_stored":6203338752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3660951552,"data_stored":5902405651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7513575424,"data_stored":12104757267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:13:02.569 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:02.569 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:02.570 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:02.570 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:02.571 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:22.472 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:13:22.470+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:13:22.473 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:13:22.470+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:13:22.475 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:13:22.474+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:13:22.475 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:13:22.474+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:13:22.490 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:13:22.486+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:13:22.491 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:13:22.486+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:13:22.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:23.091 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:13:23.092 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:13:23.117 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2164,"stamp":"2023-10-14T01:13:21.432039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26967650021,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1879620,"num_read_kb":7291674,"num_write":1842849,"num_write_kb":7370455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84299,"ondisk_log_size":84299,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18736040,"kb_used_data":14738512,"kb_used_omap":2016,"kb_used_meta":3995487,"kb_avail":264379480,"statfs":{"total":289910292480,"available":270724587520,"internally_reserved":0,"allocated":15092236288,"data_stored":24271983985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":4091379243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,12,15,6,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14782464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3778,"num_read_kb":15056,"num_write":3768,"num_write_kb":15072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002054"},"pg_stats":[{"pgid":"2.7","version":"120'6151","reported_seq":9148,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.019777+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:13:16.019777+0000","last_peered":"2023-10-14T01:13:16.019777+0000","last_clean":"2023-10-14T01:13:16.019777+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:13:16.019777+0000","last_undegraded":"2023-10-14T01:13:16.019777+0000","last_fullsized":"2023-10-14T01:13:16.019777+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6151,"log_dups_size":0,"ondisk_log_size":6151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"120'9154","reported_seq":14829,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.020098+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:13:16.020098+0000","last_peered":"2023-10-14T01:13:16.020098+0000","last_clean":"2023-10-14T01:13:16.020098+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:13:16.020098+0000","last_undegraded":"2023-10-14T01:13:16.020098+0000","last_fullsized":"2023-10-14T01:13:16.020098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9154,"log_dups_size":0,"ondisk_log_size":9154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"120'6123","reported_seq":8920,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.019368+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:13:16.019368+0000","last_peered":"2023-10-14T01:13:16.019368+0000","last_clean":"2023-10-14T01:13:16.019368+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:13:16.019368+0000","last_undegraded":"2023-10-14T01:13:16.019368+0000","last_fullsized":"2023-10-14T01:13:16.019368+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"120'15091","reported_seq":22559,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.020433+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:13:16.020433+0000","last_peered":"2023-10-14T01:13:16.020433+0000","last_clean":"2023-10-14T01:13:16.020433+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:13:16.020433+0000","last_undegraded":"2023-10-14T01:13:16.020433+0000","last_fullsized":"2023-10-14T01:13:16.020433+0000","mapping_epoch":12,"log_start":"79'5000","ondisk_log_start":"79'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"120'478205","reported_seq":951951,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:18.010836+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:13:18.010836+0000","last_peered":"2023-10-14T01:13:18.010836+0000","last_clean":"2023-10-14T01:13:18.010836+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:13:18.010836+0000","last_undegraded":"2023-10-14T01:13:18.010836+0000","last_fullsized":"2023-10-14T01:13:18.010836+0000","mapping_epoch":15,"log_start":"118'468133","ondisk_log_start":"118'468133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":485,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 92s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6965219328,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":476024,"num_read_kb":1873404,"num_write":475351,"num_write_kb":1901404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"120'3189","reported_seq":4768,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.019076+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:13:16.019076+0000","last_peered":"2023-10-14T01:13:16.019076+0000","last_clean":"2023-10-14T01:13:16.019076+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:13:16.019076+0000","last_undegraded":"2023-10-14T01:13:16.019076+0000","last_fullsized":"2023-10-14T01:13:16.019076+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"120'453868","reported_seq":904167,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:17.572352+0000","last_change":"2023-10-14T01:13:08.844914+0000","last_active":"2023-10-14T01:13:17.572352+0000","last_peered":"2023-10-14T01:13:17.572352+0000","last_clean":"2023-10-14T01:13:17.572352+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:13:17.572352+0000","last_undegraded":"2023-10-14T01:13:17.572352+0000","last_fullsized":"2023-10-14T01:13:17.572352+0000","mapping_epoch":15,"log_start":"118'443823","ondisk_log_start":"118'443823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6615166976,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":452489,"num_read_kb":1780256,"num_write":451472,"num_write_kb":1805888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2840,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.019823+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:13:16.019823+0000","last_peered":"2023-10-14T01:13:16.019823+0000","last_clean":"2023-10-14T01:13:16.019823+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:13:16.019823+0000","last_undegraded":"2023-10-14T01:13:16.019823+0000","last_fullsized":"2023-10-14T01:13:16.019823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"120'472149","reported_seq":940139,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:18.010993+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:13:18.010993+0000","last_peered":"2023-10-14T01:13:18.010993+0000","last_clean":"2023-10-14T01:13:18.010993+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:13:18.010993+0000","last_undegraded":"2023-10-14T01:13:18.010993+0000","last_fullsized":"2023-10-14T01:13:18.010993+0000","mapping_epoch":15,"log_start":"118'462128","ondisk_log_start":"118'462128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":592,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 93s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6862389248,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":470207,"num_read_kb":1850356,"num_write":469283,"num_write_kb":1877132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":2990,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.019954+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:13:16.019954+0000","last_peered":"2023-10-14T01:13:16.019954+0000","last_clean":"2023-10-14T01:13:16.019954+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:13:16.019954+0000","last_undegraded":"2023-10-14T01:13:16.019954+0000","last_fullsized":"2023-10-14T01:13:16.019954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"120'448650","reported_seq":893689,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:17.608083+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:13:17.608083+0000","last_peered":"2023-10-14T01:13:17.608083+0000","last_clean":"2023-10-14T01:13:17.608083+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:13:17.608083+0000","last_undegraded":"2023-10-14T01:13:17.608083+0000","last_fullsized":"2023-10-14T01:13:17.608083+0000","mapping_epoch":15,"log_start":"118'438623","ondisk_log_start":"118'438623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":515,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 72s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6524280851,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":447152,"num_read_kb":1758852,"num_write":445947,"num_write_kb":1783782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"120'9103","reported_seq":13194,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:16.020241+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:13:16.020241+0000","last_peered":"2023-10-14T01:13:16.020241+0000","last_clean":"2023-10-14T01:13:16.020241+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:13:16.020241+0000","last_undegraded":"2023-10-14T01:13:16.020241+0000","last_fullsized":"2023-10-14T01:13:16.020241+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":291,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-14T01:11:51.191389+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:11:51.191389+0000","last_peered":"2023-10-14T01:11:51.191389+0000","last_clean":"2023-10-14T01:11:51.191389+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:11:51.191389+0000","last_undegraded":"2023-10-14T01:11:51.191389+0000","last_fullsized":"2023-10-14T01:11:51.191389+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26967056403,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1845872,"num_read_kb":7262868,"num_write":1842053,"num_write_kb":7368206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15089152000,"data_stored":24271515686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44035,"ondisk_log_size":44035,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8425884,"kb_used_data":7367380,"kb_used_omap":1043,"kb_used_meta":1057452,"kb_avail":85945956,"statfs":{"total":96636764160,"available":88008658944,"internally_reserved":0,"allocated":7544197120,"data_stored":12134595652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1082831095},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.56299999999999994,"15min":0.46800000000000003},"min":{"1min":0.19800000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.47799999999999998,"5min":27.402000000000001,"15min":27.402000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.52500000000000002,"15min":0.56399999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.183},"max":{"1min":0.40300000000000002,"5min":27.538,"15min":27.538},"last":2.0590000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:12:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36799999999999999,"15min":0.40300000000000002},"min":{"1min":0.214,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.627,"5min":3.298,"15min":4.8739999999999997},"last":2.1899999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41399999999999998,"15min":0.45400000000000001},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.60699999999999998,"5min":3.339,"15min":12.361000000000001},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359739188,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4876984,"kb_used_data":3591152,"kb_used_omap":472,"kb_used_meta":1285351,"kb_avail":89494856,"statfs":{"total":96636764160,"available":91642732544,"internally_reserved":0,"allocated":3677339648,"data_stored":5917812936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1316200143},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,9,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:35 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.37,"15min":0.442},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16400000000000001},"max":{"1min":4.0579999999999998,"5min":4.0579999999999998,"15min":5.3300000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.53900000000000003},"min":{"1min":0.22,"5min":0.20599999999999999,"15min":0.20000000000000001},"max":{"1min":4.2329999999999997,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.249}]},{"osd":2,"last update":"Sat Oct 14 01:12:35 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.58799999999999997,"15min":0.54300000000000004},"min":{"1min":0.20100000000000001,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":4.1289999999999996,"5min":15.891,"15min":15.891},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.43099999999999999,"15min":0.58599999999999997},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":9.673},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739187,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5433172,"kb_used_data":3779980,"kb_used_omap":500,"kb_used_meta":1652683,"kb_avail":88938668,"statfs":{"total":96636764160,"available":91073196032,"internally_reserved":0,"allocated":3870699520,"data_stored":6219575397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1692348005},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,5,3,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53900000000000003,"15min":0.47499999999999998},"min":{"1min":0.188,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.50800000000000001,"15min":0.46000000000000002},"min":{"1min":0.184,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.5390000000000001,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Sat Oct 14 01:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.52300000000000002},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":3.1629999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.46700000000000003,"15min":0.53700000000000003},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.532,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3869224960,"data_stored":6218952704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3675836416,"data_stored":5917290515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7544090624,"data_stored":12135272467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:13:23.119 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:23.119 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:23.120 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:23.120 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:23.120 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:43.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:43.643 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:13:43.644 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:13:43.672 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2175,"stamp":"2023-10-14T01:13:43.435590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27004243685,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1889248,"num_read_kb":7329994,"num_write":1852258,"num_write_kb":7408091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84545,"ondisk_log_size":84545,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19029956,"kb_used_data":14813676,"kb_used_omap":2016,"kb_used_meta":4214239,"kb_avail":264085564,"statfs":{"total":289910292480,"available":270423617536,"internally_reserved":0,"allocated":15169204224,"data_stored":24348951921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":4315381291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,2,6,16,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22495232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5940,"num_read_kb":23636,"num_write":5770,"num_write_kb":23080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001939"},"pg_stats":[{"pgid":"2.7","version":"120'6181","reported_seq":9188,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.619169+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:13:41.619169+0000","last_peered":"2023-10-14T01:13:41.619169+0000","last_clean":"2023-10-14T01:13:41.619169+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:13:41.619169+0000","last_undegraded":"2023-10-14T01:13:41.619169+0000","last_fullsized":"2023-10-14T01:13:41.619169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6181,"log_dups_size":0,"ondisk_log_size":6181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"120'9197","reported_seq":14887,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.619155+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:13:41.619155+0000","last_peered":"2023-10-14T01:13:41.619155+0000","last_clean":"2023-10-14T01:13:41.619155+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:13:41.619155+0000","last_undegraded":"2023-10-14T01:13:41.619155+0000","last_fullsized":"2023-10-14T01:13:41.619155+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"120'6153","reported_seq":8960,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.618964+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:13:41.618964+0000","last_peered":"2023-10-14T01:13:41.618964+0000","last_clean":"2023-10-14T01:13:41.618964+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:13:41.618964+0000","last_undegraded":"2023-10-14T01:13:41.618964+0000","last_fullsized":"2023-10-14T01:13:41.618964+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6153,"log_dups_size":0,"ondisk_log_size":6153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"120'15162","reported_seq":22655,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.620426+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:13:41.620426+0000","last_peered":"2023-10-14T01:13:41.620426+0000","last_clean":"2023-10-14T01:13:41.620426+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:13:41.620426+0000","last_undegraded":"2023-10-14T01:13:41.620426+0000","last_fullsized":"2023-10-14T01:13:41.620426+0000","mapping_epoch":12,"log_start":"79'5100","ondisk_log_start":"79'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"120'480724","reported_seq":956907,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:43.168919+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:13:43.168919+0000","last_peered":"2023-10-14T01:13:43.168919+0000","last_clean":"2023-10-14T01:13:43.168919+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:13:43.168919+0000","last_undegraded":"2023-10-14T01:13:43.168919+0000","last_fullsized":"2023-10-14T01:13:43.168919+0000","mapping_epoch":15,"log_start":"119'470634","ondisk_log_start":"119'470634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":623,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 118s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6974689280,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":478450,"num_read_kb":1883056,"num_write":477775,"num_write_kb":1911100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"120'3209","reported_seq":4793,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:42.363331+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:13:42.363331+0000","last_peered":"2023-10-14T01:13:42.363331+0000","last_clean":"2023-10-14T01:13:42.363331+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:13:42.363331+0000","last_undegraded":"2023-10-14T01:13:42.363331+0000","last_fullsized":"2023-10-14T01:13:42.363331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3209,"log_dups_size":0,"ondisk_log_size":3209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"120'456150","reported_seq":908857,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:42.695259+0000","last_change":"2023-10-14T01:13:37.058332+0000","last_active":"2023-10-14T01:13:42.695259+0000","last_peered":"2023-10-14T01:13:42.695259+0000","last_clean":"2023-10-14T01:13:42.695259+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:13:42.695259+0000","last_undegraded":"2023-10-14T01:13:42.695259+0000","last_fullsized":"2023-10-14T01:13:42.695259+0000","mapping_epoch":15,"log_start":"119'446123","ondisk_log_start":"119'446123","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6624079872,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":454890,"num_read_kb":1789820,"num_write":453754,"num_write_kb":1815016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2850,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.621269+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:13:41.621269+0000","last_peered":"2023-10-14T01:13:41.621269+0000","last_clean":"2023-10-14T01:13:41.621269+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:13:41.621269+0000","last_undegraded":"2023-10-14T01:13:41.621269+0000","last_fullsized":"2023-10-14T01:13:41.621269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"120'474700","reported_seq":945138,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:43.166083+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:13:43.166083+0000","last_peered":"2023-10-14T01:13:43.166083+0000","last_clean":"2023-10-14T01:13:43.166083+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:13:43.166083+0000","last_undegraded":"2023-10-14T01:13:43.166083+0000","last_fullsized":"2023-10-14T01:13:43.166083+0000","mapping_epoch":15,"log_start":"119'464628","ondisk_log_start":"119'464628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":709,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 119s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6871977984,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":472642,"num_read_kb":1860028,"num_write":471747,"num_write_kb":1886988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3001,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.625120+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:13:41.625120+0000","last_peered":"2023-10-14T01:13:41.625120+0000","last_clean":"2023-10-14T01:13:41.625120+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:13:41.625120+0000","last_undegraded":"2023-10-14T01:13:41.625120+0000","last_fullsized":"2023-10-14T01:13:41.625120+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"120'451008","reported_seq":898432,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:42.690855+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:13:42.690855+0000","last_peered":"2023-10-14T01:13:42.690855+0000","last_clean":"2023-10-14T01:13:42.690855+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:13:42.690855+0000","last_undegraded":"2023-10-14T01:13:42.690855+0000","last_fullsized":"2023-10-14T01:13:42.690855+0000","mapping_epoch":15,"log_start":"119'440923","ondisk_log_start":"119'440923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":682,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 97s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6532902931,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":449518,"num_read_kb":1768284,"num_write":448186,"num_write_kb":1792738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"120'9146","reported_seq":13252,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.619492+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:13:41.619492+0000","last_peered":"2023-10-14T01:13:41.619492+0000","last_clean":"2023-10-14T01:13:41.619492+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:13:41.619492+0000","last_undegraded":"2023-10-14T01:13:41.619492+0000","last_fullsized":"2023-10-14T01:13:41.619492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9146,"log_dups_size":0,"ondisk_log_size":9146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":292,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:12:56.219253+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:12:56.219253+0000","last_peered":"2023-10-14T01:12:56.219253+0000","last_clean":"2023-10-14T01:12:56.219253+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:12:56.219253+0000","last_undegraded":"2023-10-14T01:12:56.219253+0000","last_fullsized":"2023-10-14T01:12:56.219253+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27003650067,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1855500,"num_read_kb":7301188,"num_write":1851462,"num_write_kb":7405842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15166230528,"data_stored":24348594214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40274,"ondisk_log_size":40274,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44172,"ondisk_log_size":44172,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739194,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8575812,"kb_used_data":7404924,"kb_used_omap":1043,"kb_used_meta":1169836,"kb_avail":85796028,"statfs":{"total":96636764160,"available":87855132672,"internally_reserved":0,"allocated":7582642176,"data_stored":12173040708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1197912311},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.33800000000000002,"15min":0.45800000000000002},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.52400000000000002,"5min":2.198,"15min":27.402000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.41799999999999998,"15min":0.57099999999999995},"min":{"1min":0.216,"5min":0.191,"15min":0.183},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":27.538},"last":0.23799999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:12:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36799999999999999,"15min":0.40300000000000002},"min":{"1min":0.214,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.627,"5min":3.298,"15min":4.8739999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41399999999999998,"15min":0.45400000000000001},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.60699999999999998,"5min":3.339,"15min":12.361000000000001},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4953404,"kb_used_data":3609268,"kb_used_omap":472,"kb_used_meta":1343655,"kb_avail":89418436,"statfs":{"total":96636764160,"available":91564478464,"internally_reserved":0,"allocated":3695890432,"data_stored":5936363720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1375903439},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.185,"5min":0.17599999999999999,"15min":0.16900000000000001},"max":{"1min":5.7869999999999999,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.55500000000000005},"min":{"1min":0.221,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":5.8979999999999997,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.318}]},{"osd":2,"last update":"Sat Oct 14 01:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.59299999999999997,"15min":0.51500000000000001},"min":{"1min":0.20000000000000001,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":15.891,"15min":15.891},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.44900000000000001,"15min":0.58999999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":5.835,"5min":5.835,"15min":9.673},"last":0.35199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5500740,"kb_used_data":3799484,"kb_used_omap":500,"kb_used_meta":1700747,"kb_avail":88871100,"statfs":{"total":96636764160,"available":91004006400,"internally_reserved":0,"allocated":3890671616,"data_stored":6239547493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1741565541},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,8,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.52500000000000002,"15min":0.47699999999999998},"min":{"1min":0.19500000000000001,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":1.361,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.47599999999999998},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.0830000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.34699999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.499,"15min":0.51500000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.2160000000000002,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":3.0299999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.1619999999999999,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3889246208,"data_stored":6238973952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3694362624,"data_stored":5935816723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7582621696,"data_stored":12173803539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:13:43.673 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:43.674 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:43.674 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:43.675 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:43.675 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:03.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:04.208 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:14:04.209 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:14:04.257 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2185,"stamp":"2023-10-14T01:14:03.438892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27033816805,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1897073,"num_read_kb":7361118,"num_write":1859866,"num_write_kb":7438523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84543,"ondisk_log_size":84543,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19254656,"kb_used_data":14874344,"kb_used_omap":2016,"kb_used_meta":4378271,"kb_avail":263860864,"statfs":{"total":289910292480,"available":270193524736,"internally_reserved":0,"allocated":15231328256,"data_stored":24411075953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":4483350059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,9,14,14,19,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22052864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5850,"num_read_kb":23252,"num_write":5678,"num_write_kb":22712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001983"},"pg_stats":[{"pgid":"2.7","version":"120'6215","reported_seq":9230,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.099633+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:14:02.099633+0000","last_peered":"2023-10-14T01:14:02.099633+0000","last_clean":"2023-10-14T01:14:02.099633+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:14:02.099633+0000","last_undegraded":"2023-10-14T01:14:02.099633+0000","last_fullsized":"2023-10-14T01:14:02.099633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6215,"log_dups_size":0,"ondisk_log_size":6215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"120'9246","reported_seq":14948,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.099378+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:14:02.099378+0000","last_peered":"2023-10-14T01:14:02.099378+0000","last_clean":"2023-10-14T01:14:02.099378+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:14:02.099378+0000","last_undegraded":"2023-10-14T01:14:02.099378+0000","last_fullsized":"2023-10-14T01:14:02.099378+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9246,"log_dups_size":0,"ondisk_log_size":9246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"120'6187","reported_seq":9001,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.099063+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:14:02.099063+0000","last_peered":"2023-10-14T01:14:02.099063+0000","last_clean":"2023-10-14T01:14:02.099063+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:14:02.099063+0000","last_undegraded":"2023-10-14T01:14:02.099063+0000","last_fullsized":"2023-10-14T01:14:02.099063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6187,"log_dups_size":0,"ondisk_log_size":6187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"120'15244","reported_seq":22753,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.100164+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:14:02.100164+0000","last_peered":"2023-10-14T01:14:02.100164+0000","last_clean":"2023-10-14T01:14:02.100164+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:14:02.100164+0000","last_undegraded":"2023-10-14T01:14:02.100164+0000","last_fullsized":"2023-10-14T01:14:02.100164+0000","mapping_epoch":12,"log_start":"81'5200","ondisk_log_start":"81'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"120'482736","reported_seq":960983,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:03.298785+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:14:03.298785+0000","last_peered":"2023-10-14T01:14:03.298785+0000","last_clean":"2023-10-14T01:14:03.298785+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:14:03.298785+0000","last_undegraded":"2023-10-14T01:14:03.298785+0000","last_fullsized":"2023-10-14T01:14:03.298785+0000","mapping_epoch":15,"log_start":"119'472734","ondisk_log_start":"119'472734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":730,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6982242304,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":480501,"num_read_kb":1891224,"num_write":479715,"num_write_kb":1918860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"120'3220","reported_seq":4808,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:01.982426+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:14:01.982426+0000","last_peered":"2023-10-14T01:14:01.982426+0000","last_clean":"2023-10-14T01:14:01.982426+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:14:01.982426+0000","last_undegraded":"2023-10-14T01:14:01.982426+0000","last_fullsized":"2023-10-14T01:14:01.982426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"120'457952","reported_seq":912615,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.845190+0000","last_change":"2023-10-14T01:14:02.051107+0000","last_active":"2023-10-14T01:14:02.845190+0000","last_peered":"2023-10-14T01:14:02.845190+0000","last_clean":"2023-10-14T01:14:02.845190+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:14:02.845190+0000","last_undegraded":"2023-10-14T01:14:02.845190+0000","last_fullsized":"2023-10-14T01:14:02.845190+0000","mapping_epoch":15,"log_start":"119'447923","ondisk_log_start":"119'447923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6631137280,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":456835,"num_read_kb":1797552,"num_write":455556,"num_write_kb":1822224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2859,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.172965+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:14:02.172965+0000","last_peered":"2023-10-14T01:14:02.172965+0000","last_clean":"2023-10-14T01:14:02.172965+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:14:02.172965+0000","last_undegraded":"2023-10-14T01:14:02.172965+0000","last_fullsized":"2023-10-14T01:14:02.172965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"120'476737","reported_seq":949182,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:03.297564+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:14:03.297564+0000","last_peered":"2023-10-14T01:14:03.297564+0000","last_clean":"2023-10-14T01:14:03.297564+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:14:03.297564+0000","last_undegraded":"2023-10-14T01:14:03.297564+0000","last_fullsized":"2023-10-14T01:14:03.297564+0000","mapping_epoch":15,"log_start":"119'466728","ondisk_log_start":"119'466728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":795,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 139s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6879567872,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":474636,"num_read_kb":1867964,"num_write":473726,"num_write_kb":1894904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3009,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.099436+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:14:02.099436+0000","last_peered":"2023-10-14T01:14:02.099436+0000","last_clean":"2023-10-14T01:14:02.099436+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:14:02.099436+0000","last_undegraded":"2023-10-14T01:14:02.099436+0000","last_fullsized":"2023-10-14T01:14:02.099436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"120'452996","reported_seq":902268,"reported_epoch":120,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:02.854890+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:14:02.854890+0000","last_peered":"2023-10-14T01:14:02.854890+0000","last_clean":"2023-10-14T01:14:02.854890+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:14:02.854890+0000","last_undegraded":"2023-10-14T01:14:02.854890+0000","last_fullsized":"2023-10-14T01:14:02.854890+0000","mapping_epoch":15,"log_start":"119'442923","ondisk_log_start":"119'442923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":831,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 117s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6540275731,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":451353,"num_read_kb":1775572,"num_write":450073,"num_write_kb":1800286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"120'9195","reported_seq":13313,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.099475+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:14:02.099475+0000","last_peered":"2023-10-14T01:14:02.099475+0000","last_clean":"2023-10-14T01:14:02.099475+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:14:02.099475+0000","last_undegraded":"2023-10-14T01:14:02.099475+0000","last_fullsized":"2023-10-14T01:14:02.099475+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9195,"log_dups_size":0,"ondisk_log_size":9195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":292,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:12:56.219253+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:12:56.219253+0000","last_peered":"2023-10-14T01:12:56.219253+0000","last_clean":"2023-10-14T01:12:56.219253+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:12:56.219253+0000","last_undegraded":"2023-10-14T01:12:56.219253+0000","last_fullsized":"2023-10-14T01:12:56.219253+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27033223187,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1863325,"num_read_kb":7332312,"num_write":1859070,"num_write_kb":7436274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15228465152,"data_stored":24410828838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40113,"ondisk_log_size":40113,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44331,"ondisk_log_size":44331,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739198,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8701968,"kb_used_data":7435080,"kb_used_omap":1043,"kb_used_meta":1265836,"kb_avail":85669872,"statfs":{"total":96636764160,"available":87725948928,"internally_reserved":0,"allocated":7613521920,"data_stored":12203920452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1296216311},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.33800000000000002,"15min":0.45800000000000002},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.52400000000000002,"5min":2.198,"15min":27.402000000000001},"last":0.755},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.41799999999999998,"15min":0.57099999999999995},"min":{"1min":0.216,"5min":0.191,"15min":0.183},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":27.538},"last":0.58799999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:13:46 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.48099999999999998,"15min":0.439},"min":{"1min":0.27000000000000002,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.31},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.46200000000000002,"15min":0.46600000000000003},"min":{"1min":0.214,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.1699999999999999,"5min":4.1699999999999999,"15min":12.361000000000001},"last":0.626}]}]},{"osd":1,"up_from":8,"seq":34359739197,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000040,"kb_used_data":3623968,"kb_used_omap":472,"kb_used_meta":1375591,"kb_avail":89371800,"statfs":{"total":96636764160,"available":91516723200,"internally_reserved":0,"allocated":3710943232,"data_stored":5951416520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1408605903},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.185,"5min":0.17599999999999999,"15min":0.16900000000000001},"max":{"1min":5.7869999999999999,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.55500000000000005},"min":{"1min":0.221,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":5.8979999999999997,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.246}]},{"osd":2,"last update":"Sat Oct 14 01:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.59299999999999997,"15min":0.51500000000000001},"min":{"1min":0.20000000000000001,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":15.891,"15min":15.891},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.44900000000000001,"15min":0.58999999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":5.835,"5min":5.835,"15min":9.673},"last":0.48599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739196,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5552648,"kb_used_data":3815296,"kb_used_omap":500,"kb_used_meta":1736843,"kb_avail":88819192,"statfs":{"total":96636764160,"available":90950852608,"internally_reserved":0,"allocated":3906863104,"data_stored":6255738981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1778527845},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,8,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.52500000000000002,"15min":0.47699999999999998},"min":{"1min":0.19500000000000001,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":1.361,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.47599999999999998},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.0830000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.499,"15min":0.51500000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.2160000000000002,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.1619999999999999,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3905294336,"data_stored":6255022080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3709468672,"data_stored":5950922771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7613702144,"data_stored":12204883987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:14:04.259 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:04.259 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:04.260 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:04.260 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:04.261 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:07.553 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:14:07.553 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:14:07.554 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:14:07.554 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:14:07.554 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:14:07.554 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:14:07.555 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:14:07.555 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:14:07.555 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:14:07.556 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:14:07.557 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:14:07.557 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:14:07.557 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:14:07.549+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:14:24.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:24.843 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:14:24.844 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:14:24.871 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2196,"stamp":"2023-10-14T01:14:23.442252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27063430885,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1904779,"num_read_kb":7391754,"num_write":1867494,"num_write_kb":7469035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84682,"ondisk_log_size":84682,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19485828,"kb_used_data":14936684,"kb_used_omap":2016,"kb_used_meta":4547103,"kb_avail":263629692,"statfs":{"total":289910292480,"available":269956804608,"internally_reserved":0,"allocated":15295164416,"data_stored":24474889202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":4656234027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,5,7,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14753792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3863,"num_read_kb":15356,"num_write":3805,"num_write_kb":15220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001752"},"pg_stats":[{"pgid":"2.7","version":"121'6241","reported_seq":9265,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.575295+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:14:22.575295+0000","last_peered":"2023-10-14T01:14:22.575295+0000","last_clean":"2023-10-14T01:14:22.575295+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:14:22.575295+0000","last_undegraded":"2023-10-14T01:14:22.575295+0000","last_fullsized":"2023-10-14T01:14:22.575295+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6241,"log_dups_size":0,"ondisk_log_size":6241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"121'9285","reported_seq":15000,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.575144+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:14:22.575144+0000","last_peered":"2023-10-14T01:14:22.575144+0000","last_clean":"2023-10-14T01:14:22.575144+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:14:22.575144+0000","last_undegraded":"2023-10-14T01:14:22.575144+0000","last_fullsized":"2023-10-14T01:14:22.575144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"121'6213","reported_seq":9036,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.574951+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:14:22.574951+0000","last_peered":"2023-10-14T01:14:22.574951+0000","last_clean":"2023-10-14T01:14:22.574951+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:14:22.574951+0000","last_undegraded":"2023-10-14T01:14:22.574951+0000","last_fullsized":"2023-10-14T01:14:22.574951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"121'15309","reported_seq":22839,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.575699+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:14:22.575699+0000","last_peered":"2023-10-14T01:14:22.575699+0000","last_clean":"2023-10-14T01:14:22.575699+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:14:22.575699+0000","last_undegraded":"2023-10-14T01:14:22.575699+0000","last_fullsized":"2023-10-14T01:14:22.575699+0000","mapping_epoch":12,"log_start":"81'5300","ondisk_log_start":"81'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"121'484740","reported_seq":964977,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:23.377210+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:14:23.377210+0000","last_peered":"2023-10-14T01:14:23.377210+0000","last_clean":"2023-10-14T01:14:23.377210+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:14:23.377210+0000","last_undegraded":"2023-10-14T01:14:23.377210+0000","last_fullsized":"2023-10-14T01:14:23.377210+0000","mapping_epoch":15,"log_start":"119'474734","ondisk_log_start":"119'474734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":826,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 158s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6989819904,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":482479,"num_read_kb":1899096,"num_write":481653,"num_write_kb":1926612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"121'3234","reported_seq":4827,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:21.583608+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:14:21.583608+0000","last_peered":"2023-10-14T01:14:21.583608+0000","last_clean":"2023-10-14T01:14:21.583608+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:14:21.583608+0000","last_undegraded":"2023-10-14T01:14:21.583608+0000","last_fullsized":"2023-10-14T01:14:21.583608+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"121'459837","reported_seq":916424,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.956090+0000","last_change":"2023-10-14T01:14:20.161541+0000","last_active":"2023-10-14T01:14:22.956090+0000","last_peered":"2023-10-14T01:14:22.956090+0000","last_clean":"2023-10-14T01:14:22.956090+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:14:22.956090+0000","last_undegraded":"2023-10-14T01:14:22.956090+0000","last_fullsized":"2023-10-14T01:14:22.956090+0000","mapping_epoch":15,"log_start":"119'449823","ondisk_log_start":"119'449823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6638440448,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":458746,"num_read_kb":1805152,"num_write":457441,"num_write_kb":1829764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2866,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:17.455388+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:14:17.455388+0000","last_peered":"2023-10-14T01:14:17.455388+0000","last_clean":"2023-10-14T01:14:17.455388+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:14:17.455388+0000","last_undegraded":"2023-10-14T01:14:17.455388+0000","last_fullsized":"2023-10-14T01:14:17.455388+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"121'478666","reported_seq":953017,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:23.373696+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:14:23.373696+0000","last_peered":"2023-10-14T01:14:23.373696+0000","last_clean":"2023-10-14T01:14:23.373696+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:14:23.373696+0000","last_undegraded":"2023-10-14T01:14:23.373696+0000","last_fullsized":"2023-10-14T01:14:23.373696+0000","mapping_epoch":15,"log_start":"119'468628","ondisk_log_start":"119'468628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":908,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 159s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6886699008,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":476532,"num_read_kb":1875508,"num_write":475579,"num_write_kb":1902316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3016,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:17.455199+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:14:17.455199+0000","last_peered":"2023-10-14T01:14:17.455199+0000","last_clean":"2023-10-14T01:14:17.455199+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:14:17.455199+0000","last_undegraded":"2023-10-14T01:14:17.455199+0000","last_fullsized":"2023-10-14T01:14:17.455199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"121'455008","reported_seq":906214,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:22.962127+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:14:22.962127+0000","last_peered":"2023-10-14T01:14:22.962127+0000","last_clean":"2023-10-14T01:14:22.962127+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:14:22.962127+0000","last_undegraded":"2023-10-14T01:14:22.962127+0000","last_fullsized":"2023-10-14T01:14:22.962127+0000","mapping_epoch":15,"log_start":"119'444923","ondisk_log_start":"119'444923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":920,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 137s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6547877907,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":453274,"num_read_kb":1783192,"num_write":452025,"num_write_kb":1808094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"121'9234","reported_seq":13365,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.575465+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:14:22.575465+0000","last_peered":"2023-10-14T01:14:22.575465+0000","last_clean":"2023-10-14T01:14:22.575465+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:14:22.575465+0000","last_undegraded":"2023-10-14T01:14:22.575465+0000","last_fullsized":"2023-10-14T01:14:22.575465+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":293,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:37.668343+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:13:37.668343+0000","last_peered":"2023-10-14T01:13:37.668343+0000","last_clean":"2023-10-14T01:13:37.668343+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:13:37.668343+0000","last_undegraded":"2023-10-14T01:13:37.668343+0000","last_fullsized":"2023-10-14T01:13:37.668343+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27062837267,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1871031,"num_read_kb":7362948,"num_write":1866698,"num_write_kb":7466786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15290966016,"data_stored":24473329702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40143,"ondisk_log_size":40143,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44440,"ondisk_log_size":44440,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739202,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8823860,"kb_used_data":7466796,"kb_used_omap":1043,"kb_used_meta":1356012,"kb_avail":85547980,"statfs":{"total":96636764160,"available":87601131520,"internally_reserved":0,"allocated":7645999104,"data_stored":12236389999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1388556535},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:26 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.33800000000000002,"15min":0.45800000000000002},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.52400000000000002,"5min":2.198,"15min":27.402000000000001},"last":0.249},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.41799999999999998,"15min":0.57099999999999995},"min":{"1min":0.216,"5min":0.191,"15min":0.183},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":27.538},"last":0.375}]},{"osd":1,"last update":"Sat Oct 14 01:13:46 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.48099999999999998,"15min":0.439},"min":{"1min":0.27000000000000002,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.46200000000000002,"15min":0.46600000000000003},"min":{"1min":0.214,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.1699999999999999,"5min":4.1699999999999999,"15min":12.361000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5033388,"kb_used_data":3639332,"kb_used_omap":472,"kb_used_meta":1393575,"kb_avail":89338452,"statfs":{"total":96636764160,"available":91482574848,"internally_reserved":0,"allocated":3726675968,"data_stored":5967141619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1427021519},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.185,"5min":0.17599999999999999,"15min":0.16900000000000001},"max":{"1min":5.7869999999999999,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.55500000000000005},"min":{"1min":0.221,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":5.8979999999999997,"5min":6.9199999999999999,"15min":10.122999999999999},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.59299999999999997,"15min":0.51500000000000001},"min":{"1min":0.20000000000000001,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":15.891,"15min":15.891},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.44900000000000001,"15min":0.58999999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":5.835,"5min":5.835,"15min":9.673},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5628580,"kb_used_data":3830556,"kb_used_omap":500,"kb_used_meta":1797515,"kb_avail":88743260,"statfs":{"total":96636764160,"available":90873098240,"internally_reserved":0,"allocated":3922489344,"data_stored":6271357584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1840655973},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.52500000000000002,"15min":0.47699999999999998},"min":{"1min":0.19500000000000001,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":1.361,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.47599999999999998},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.0830000000000002,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.499,"15min":0.51500000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.2160000000000002,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":3.1619999999999999,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3920826368,"data_stored":6270554112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3725168640,"data_stored":5966622739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7644971008,"data_stored":12236152851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:14:24.873 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:24.873 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:24.874 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:24.874 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:24.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:14:25.608 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:14:25.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:14:26.275 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:14:26.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:14:26.874 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:14:26.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:14:27.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-10-14T01:14:27.345+0000 7ff49659c640 0 --2- 172.21.15.81:0/4109337074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff480003ab0 0x7ff480003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T01:14:27.683 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:14:27.712 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:47.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:48.251 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:14:48.252 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:14:48.324 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2208,"stamp":"2023-10-14T01:14:47.446176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27092393701,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1912054,"num_read_kb":7420670,"num_write":1874919,"num_write_kb":7498735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84937,"ondisk_log_size":84937,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19672960,"kb_used_data":14995816,"kb_used_omap":2016,"kb_used_meta":4675103,"kb_avail":263442560,"statfs":{"total":289910292480,"available":269765181440,"internally_reserved":0,"allocated":15355715584,"data_stored":24535440370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":4787306027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,7,9,6,13,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14966784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3787,"num_read_kb":15056,"num_write":3849,"num_write_kb":15396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002023"},"pg_stats":[{"pgid":"2.7","version":"121'6265","reported_seq":9295,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:42.850132+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:14:42.850132+0000","last_peered":"2023-10-14T01:14:42.850132+0000","last_clean":"2023-10-14T01:14:42.850132+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:14:42.850132+0000","last_undegraded":"2023-10-14T01:14:42.850132+0000","last_fullsized":"2023-10-14T01:14:42.850132+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6265,"log_dups_size":0,"ondisk_log_size":6265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"121'9321","reported_seq":15045,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:42.864319+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:14:42.864319+0000","last_peered":"2023-10-14T01:14:42.864319+0000","last_clean":"2023-10-14T01:14:42.864319+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:14:42.864319+0000","last_undegraded":"2023-10-14T01:14:42.864319+0000","last_fullsized":"2023-10-14T01:14:42.864319+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"121'6237","reported_seq":9067,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:42.829717+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:14:42.829717+0000","last_peered":"2023-10-14T01:14:42.829717+0000","last_clean":"2023-10-14T01:14:42.829717+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:14:42.829717+0000","last_undegraded":"2023-10-14T01:14:42.829717+0000","last_fullsized":"2023-10-14T01:14:42.829717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6237,"log_dups_size":0,"ondisk_log_size":6237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"121'15369","reported_seq":22915,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:42.898740+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:14:42.898740+0000","last_peered":"2023-10-14T01:14:42.898740+0000","last_clean":"2023-10-14T01:14:42.898740+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:14:42.898740+0000","last_undegraded":"2023-10-14T01:14:42.898740+0000","last_fullsized":"2023-10-14T01:14:42.898740+0000","mapping_epoch":12,"log_start":"81'5300","ondisk_log_start":"81'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"121'486711","reported_seq":968825,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:43.508581+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:14:43.508581+0000","last_peered":"2023-10-14T01:14:43.508581+0000","last_clean":"2023-10-14T01:14:43.508581+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:14:43.508581+0000","last_undegraded":"2023-10-14T01:14:43.508581+0000","last_fullsized":"2023-10-14T01:14:43.508581+0000","mapping_epoch":15,"log_start":"120'476634","ondisk_log_start":"120'476634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":932,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 178s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":6997237760,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":484347,"num_read_kb":1906500,"num_write":483551,"num_write_kb":1934204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"121'3251","reported_seq":4849,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:42.076718+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:14:42.076718+0000","last_peered":"2023-10-14T01:14:42.076718+0000","last_clean":"2023-10-14T01:14:42.076718+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:14:42.076718+0000","last_undegraded":"2023-10-14T01:14:42.076718+0000","last_fullsized":"2023-10-14T01:14:42.076718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3251,"log_dups_size":0,"ondisk_log_size":3251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"121'461629","reported_seq":920047,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:43.021240+0000","last_change":"2023-10-14T01:14:42.426727+0000","last_active":"2023-10-14T01:14:43.021240+0000","last_peered":"2023-10-14T01:14:43.021240+0000","last_clean":"2023-10-14T01:14:43.021240+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:14:43.021240+0000","last_undegraded":"2023-10-14T01:14:43.021240+0000","last_fullsized":"2023-10-14T01:14:43.021240+0000","mapping_epoch":15,"log_start":"119'451623","ondisk_log_start":"119'451623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6645473280,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":460572,"num_read_kb":1812420,"num_write":459233,"num_write_kb":1836932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2874,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:37.938314+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:14:37.938314+0000","last_peered":"2023-10-14T01:14:37.938314+0000","last_clean":"2023-10-14T01:14:37.938314+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:14:37.938314+0000","last_undegraded":"2023-10-14T01:14:37.938314+0000","last_fullsized":"2023-10-14T01:14:37.938314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"121'480684","reported_seq":956896,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:43.503249+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:14:43.503249+0000","last_peered":"2023-10-14T01:14:43.503249+0000","last_clean":"2023-10-14T01:14:43.503249+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:14:43.503249+0000","last_undegraded":"2023-10-14T01:14:43.503249+0000","last_fullsized":"2023-10-14T01:14:43.503249+0000","mapping_epoch":15,"log_start":"120'470628","ondisk_log_start":"120'470628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1004,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 179s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6894301184,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":478382,"num_read_kb":1882868,"num_write":477530,"num_write_kb":1910120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3024,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:37.937175+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:14:37.937175+0000","last_peered":"2023-10-14T01:14:37.937175+0000","last_clean":"2023-10-14T01:14:37.937175+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:14:37.937175+0000","last_undegraded":"2023-10-14T01:14:37.937175+0000","last_fullsized":"2023-10-14T01:14:37.937175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"121'456887","reported_seq":909836,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:43.020116+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:14:43.020116+0000","last_peered":"2023-10-14T01:14:43.020116+0000","last_clean":"2023-10-14T01:14:43.020116+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:14:43.020116+0000","last_undegraded":"2023-10-14T01:14:43.020116+0000","last_fullsized":"2023-10-14T01:14:43.020116+0000","mapping_epoch":15,"log_start":"120'446825","ondisk_log_start":"120'446825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1060,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 157s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6554787859,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":455005,"num_read_kb":1790076,"num_write":453809,"num_write_kb":1815230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"121'9270","reported_seq":13411,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:42.923793+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:14:42.923793+0000","last_peered":"2023-10-14T01:14:42.923793+0000","last_clean":"2023-10-14T01:14:42.923793+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:14:42.923793+0000","last_undegraded":"2023-10-14T01:14:42.923793+0000","last_fullsized":"2023-10-14T01:14:42.923793+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9270,"log_dups_size":0,"ondisk_log_size":9270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":293,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:37.668343+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:13:37.668343+0000","last_peered":"2023-10-14T01:13:37.668343+0000","last_clean":"2023-10-14T01:13:37.668343+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:13:37.668343+0000","last_undegraded":"2023-10-14T01:13:37.668343+0000","last_fullsized":"2023-10-14T01:13:37.668343+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27091800083,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1878306,"num_read_kb":7391864,"num_write":1874123,"num_write_kb":7496486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15351721984,"data_stored":24534085670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40201,"ondisk_log_size":40201,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44637,"ondisk_log_size":44637,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739206,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8872892,"kb_used_data":7496116,"kb_used_omap":1043,"kb_used_meta":1375724,"kb_avail":85498948,"statfs":{"total":96636764160,"available":87550922752,"internally_reserved":0,"allocated":7676022784,"data_stored":12266413679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1408741623},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:29 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.443},"min":{"1min":0.217,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.755,"5min":2.198,"15min":27.402000000000001},"last":0.216},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.46200000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.183},"max":{"1min":3.7349999999999999,"5min":5.4050000000000002,"15min":27.538},"last":0.38}]},{"osd":1,"last update":"Sat Oct 14 01:13:46 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.48099999999999998,"15min":0.439},"min":{"1min":0.27000000000000002,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.436},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.46200000000000002,"15min":0.46600000000000003},"min":{"1min":0.214,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.1699999999999999,"5min":4.1699999999999999,"15min":12.361000000000001},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359739205,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5119984,"kb_used_data":3653736,"kb_used_omap":472,"kb_used_meta":1465767,"kb_avail":89251856,"statfs":{"total":96636764160,"available":91393900544,"internally_reserved":0,"allocated":3741425664,"data_stored":5981891315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1500946127},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.495,"15min":0.439},"min":{"1min":0.214,"5min":0.182,"15min":0.16900000000000001},"max":{"1min":2.738,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.53600000000000003},"min":{"1min":0.246,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":2.6379999999999999,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41599999999999998,"15min":0.50900000000000001},"min":{"1min":0.214,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":0.48499999999999999,"5min":4.1289999999999996,"15min":15.891},"last":0.375},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.53700000000000003,"15min":0.56200000000000006},"min":{"1min":0.254,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":2.6720000000000002,"5min":5.835,"15min":7.1600000000000001},"last":0.42899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5680084,"kb_used_data":3845964,"kb_used_omap":500,"kb_used_meta":1833611,"kb_avail":88691756,"statfs":{"total":96636764160,"available":90820358144,"internally_reserved":0,"allocated":3938267136,"data_stored":6287135376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1877618277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,10,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.47799999999999998},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":4.1539999999999999,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.47199999999999998},"min":{"1min":0.20999999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48899999999999999,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:14:36 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.49099999999999999},"min":{"1min":0.20100000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.0299999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.496},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":1.1259999999999999,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.372}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3936600064,"data_stored":6286327808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3739828224,"data_stored":5981282323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7675293696,"data_stored":12266475539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:14:48.325 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:48.326 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:48.326 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:48.327 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:48.327 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:08.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:08.866 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:15:08.866 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:15:08.894 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2218,"stamp":"2023-10-14T01:15:07.452043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27122020069,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1919365,"num_read_kb":7449742,"num_write":1882520,"num_write_kb":7529139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85056,"ondisk_log_size":85056,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19891936,"kb_used_data":15056648,"kb_used_omap":2016,"kb_used_meta":4833247,"kb_avail":263223584,"statfs":{"total":289910292480,"available":269540950016,"internally_reserved":0,"allocated":15418007552,"data_stored":24597732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":4949245483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,7,5,14,11,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14983168,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3620,"num_read_kb":14408,"num_write":3847,"num_write_kb":15388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002033"},"pg_stats":[{"pgid":"2.7","version":"121'6287","reported_seq":9326,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.986362+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:15:02.986362+0000","last_peered":"2023-10-14T01:15:02.986362+0000","last_clean":"2023-10-14T01:15:02.986362+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:15:02.986362+0000","last_undegraded":"2023-10-14T01:15:02.986362+0000","last_fullsized":"2023-10-14T01:15:02.986362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"121'9354","reported_seq":15091,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.995356+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:15:02.995356+0000","last_peered":"2023-10-14T01:15:02.995356+0000","last_clean":"2023-10-14T01:15:02.995356+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:15:02.995356+0000","last_undegraded":"2023-10-14T01:15:02.995356+0000","last_fullsized":"2023-10-14T01:15:02.995356+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"121'6259","reported_seq":9097,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.979822+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:15:02.979822+0000","last_peered":"2023-10-14T01:15:02.979822+0000","last_clean":"2023-10-14T01:15:02.979822+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:15:02.979822+0000","last_undegraded":"2023-10-14T01:15:02.979822+0000","last_fullsized":"2023-10-14T01:15:02.979822+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"121'15424","reported_seq":22990,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:03.009045+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:15:03.009045+0000","last_peered":"2023-10-14T01:15:03.009045+0000","last_clean":"2023-10-14T01:15:03.009045+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:15:03.009045+0000","last_undegraded":"2023-10-14T01:15:03.009045+0000","last_fullsized":"2023-10-14T01:15:03.009045+0000","mapping_epoch":12,"log_start":"81'5400","ondisk_log_start":"81'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"121'488757","reported_seq":972726,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:03.640167+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:15:03.640167+0000","last_peered":"2023-10-14T01:15:03.640167+0000","last_clean":"2023-10-14T01:15:03.640167+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:15:03.640167+0000","last_undegraded":"2023-10-14T01:15:03.640167+0000","last_fullsized":"2023-10-14T01:15:03.640167+0000","mapping_epoch":15,"log_start":"120'478734","ondisk_log_start":"120'478734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1040,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7004930048,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":486186,"num_read_kb":1913832,"num_write":485522,"num_write_kb":1942088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"121'3266","reported_seq":4868,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:59.997189+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:14:59.997189+0000","last_peered":"2023-10-14T01:14:59.997189+0000","last_clean":"2023-10-14T01:14:59.997189+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:14:59.997189+0000","last_undegraded":"2023-10-14T01:14:59.997189+0000","last_fullsized":"2023-10-14T01:14:59.997189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3266,"log_dups_size":0,"ondisk_log_size":3266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"121'463520","reported_seq":923751,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:03.166124+0000","last_change":"2023-10-14T01:15:02.744980+0000","last_active":"2023-10-14T01:15:03.166124+0000","last_peered":"2023-10-14T01:15:03.166124+0000","last_clean":"2023-10-14T01:15:03.166124+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:15:03.166124+0000","last_undegraded":"2023-10-14T01:15:03.166124+0000","last_fullsized":"2023-10-14T01:15:03.166124+0000","mapping_epoch":15,"log_start":"120'453423","ondisk_log_start":"120'453423","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6652887040,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":462379,"num_read_kb":1819608,"num_write":461124,"num_write_kb":1844496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2882,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:58.416270+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:14:58.416270+0000","last_peered":"2023-10-14T01:14:58.416270+0000","last_clean":"2023-10-14T01:14:58.416270+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:14:58.416270+0000","last_undegraded":"2023-10-14T01:14:58.416270+0000","last_fullsized":"2023-10-14T01:14:58.416270+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"121'482681","reported_seq":960793,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:03.637456+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:15:03.637456+0000","last_peered":"2023-10-14T01:15:03.637456+0000","last_clean":"2023-10-14T01:15:03.637456+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:15:03.637456+0000","last_undegraded":"2023-10-14T01:15:03.637456+0000","last_fullsized":"2023-10-14T01:15:03.637456+0000","mapping_epoch":15,"log_start":"120'472628","ondisk_log_start":"120'472628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1119,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 199s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6901764096,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":480270,"num_read_kb":1890360,"num_write":479450,"num_write_kb":1917800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3032,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:58.421415+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:14:58.421415+0000","last_peered":"2023-10-14T01:14:58.421415+0000","last_clean":"2023-10-14T01:14:58.421415+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:14:58.421415+0000","last_undegraded":"2023-10-14T01:14:58.421415+0000","last_fullsized":"2023-10-14T01:14:58.421415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"121'458792","reported_seq":913528,"reported_epoch":121,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:03.187031+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:15:03.187031+0000","last_peered":"2023-10-14T01:15:03.187031+0000","last_clean":"2023-10-14T01:15:03.187031+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:15:03.187031+0000","last_undegraded":"2023-10-14T01:15:03.187031+0000","last_fullsized":"2023-10-14T01:15:03.187031+0000","mapping_epoch":15,"log_start":"120'448725","ondisk_log_start":"120'448725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1188,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 177s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6561845267,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":456782,"num_read_kb":1797136,"num_write":455628,"num_write_kb":1822506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"121'9303","reported_seq":13456,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:03.025669+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:15:03.025669+0000","last_peered":"2023-10-14T01:15:03.025669+0000","last_clean":"2023-10-14T01:15:03.025669+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:15:03.025669+0000","last_undegraded":"2023-10-14T01:15:03.025669+0000","last_fullsized":"2023-10-14T01:15:03.025669+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9303,"log_dups_size":0,"ondisk_log_size":9303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":293,"reported_epoch":120,"state":"active+clean","last_fresh":"2023-10-14T01:13:37.668343+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:13:37.668343+0000","last_peered":"2023-10-14T01:13:37.668343+0000","last_clean":"2023-10-14T01:13:37.668343+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:13:37.668343+0000","last_undegraded":"2023-10-14T01:13:37.668343+0000","last_fullsized":"2023-10-14T01:13:37.668343+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27121426451,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1885617,"num_read_kb":7420936,"num_write":1881724,"num_write_kb":7526890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15413915648,"data_stored":24596279334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44717,"ondisk_log_size":44717,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739210,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8989308,"kb_used_data":7526516,"kb_used_omap":1043,"kb_used_meta":1461740,"kb_avail":85382532,"statfs":{"total":96636764160,"available":87431712768,"internally_reserved":0,"allocated":7707152384,"data_stored":12297543279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1496822007},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:29 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.443},"min":{"1min":0.217,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.755,"5min":2.198,"15min":27.402000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.46200000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.183},"max":{"1min":3.7349999999999999,"5min":5.4050000000000002,"15min":27.538},"last":0.25}]},{"osd":1,"last update":"Sat Oct 14 01:14:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.47499999999999998,"15min":0.42499999999999999},"min":{"1min":0.23000000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.55700000000000005,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.46600000000000003,"15min":0.45400000000000001},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":0.626,"5min":4.1699999999999999,"15min":12.361000000000001},"last":0.42399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739209,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152916,"kb_used_data":3668684,"kb_used_omap":472,"kb_used_meta":1483751,"kb_avail":89218924,"statfs":{"total":96636764160,"available":91360178176,"internally_reserved":0,"allocated":3756732416,"data_stored":5997198067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1519361743},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.495,"15min":0.439},"min":{"1min":0.214,"5min":0.182,"15min":0.16900000000000001},"max":{"1min":2.738,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.53600000000000003},"min":{"1min":0.246,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":2.6379999999999999,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41599999999999998,"15min":0.50900000000000001},"min":{"1min":0.214,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":0.48499999999999999,"5min":4.1289999999999996,"15min":15.891},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.53700000000000003,"15min":0.56200000000000006},"min":{"1min":0.254,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":2.6720000000000002,"5min":5.835,"15min":7.1600000000000001},"last":0.39900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5749712,"kb_used_data":3861448,"kb_used_omap":500,"kb_used_meta":1887755,"kb_avail":88622128,"statfs":{"total":96636764160,"available":90749059072,"internally_reserved":0,"allocated":3954122752,"data_stored":6302990992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1933061733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,1,12,8,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.47799999999999998},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":4.1539999999999999,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.47199999999999998},"min":{"1min":0.20999999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48899999999999999,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":3.5739999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:14:36 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.49099999999999999},"min":{"1min":0.20100000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.0299999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.316},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":1.1259999999999999,"5min":4.9690000000000003,"15min":18.966999999999999},"last":3.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3952529408,"data_stored":6302257152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3755028480,"data_stored":5996482579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7706357760,"data_stored":12297539603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:15:08.895 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:08.896 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:08.896 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:08.897 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:08.897 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:13.693 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:15:13.688+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:15:28.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:29.495 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:15:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:15:29.534 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2230,"stamp":"2023-10-14T01:15:29.455865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27158998757,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1928229,"num_read_kb":7485058,"num_write":1891983,"num_write_kb":7566991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85263,"ondisk_log_size":85263,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20158188,"kb_used_data":15131916,"kb_used_omap":2016,"kb_used_meta":5024223,"kb_avail":262957332,"statfs":{"total":289910292480,"available":269268307968,"internally_reserved":0,"allocated":15495081984,"data_stored":24674783730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5144804907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,10,6,10,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15015936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3562,"num_read_kb":14200,"num_write":3852,"num_write_kb":15408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001706"},"pg_stats":[{"pgid":"2.7","version":"122'6317","reported_seq":9367,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.954012+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:15:27.954012+0000","last_peered":"2023-10-14T01:15:27.954012+0000","last_clean":"2023-10-14T01:15:27.954012+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:15:27.954012+0000","last_undegraded":"2023-10-14T01:15:27.954012+0000","last_fullsized":"2023-10-14T01:15:27.954012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'9399","reported_seq":15152,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.969501+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:15:27.969501+0000","last_peered":"2023-10-14T01:15:27.969501+0000","last_clean":"2023-10-14T01:15:27.969501+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:15:27.969501+0000","last_undegraded":"2023-10-14T01:15:27.969501+0000","last_fullsized":"2023-10-14T01:15:27.969501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9399,"log_dups_size":0,"ondisk_log_size":9399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'6289","reported_seq":9138,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.968354+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:15:27.968354+0000","last_peered":"2023-10-14T01:15:27.968354+0000","last_clean":"2023-10-14T01:15:27.968354+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:15:27.968354+0000","last_undegraded":"2023-10-14T01:15:27.968354+0000","last_fullsized":"2023-10-14T01:15:27.968354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'15499","reported_seq":23091,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.997022+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:15:27.997022+0000","last_peered":"2023-10-14T01:15:27.997022+0000","last_clean":"2023-10-14T01:15:27.997022+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:15:27.997022+0000","last_undegraded":"2023-10-14T01:15:27.997022+0000","last_fullsized":"2023-10-14T01:15:27.997022+0000","mapping_epoch":12,"log_start":"81'5400","ondisk_log_start":"81'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'491349","reported_seq":977652,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:28.771214+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:15:28.771214+0000","last_peered":"2023-10-14T01:15:28.771214+0000","last_clean":"2023-10-14T01:15:28.771214+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:15:28.771214+0000","last_undegraded":"2023-10-14T01:15:28.771214+0000","last_fullsized":"2023-10-14T01:15:28.771214+0000","mapping_epoch":15,"log_start":"120'481334","ondisk_log_start":"120'481334","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1171,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 223s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7014699008,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":488506,"num_read_kb":1923088,"num_write":488026,"num_write_kb":1952104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"121'3281","reported_seq":4888,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:26.170022+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:15:26.170022+0000","last_peered":"2023-10-14T01:15:26.170022+0000","last_clean":"2023-10-14T01:15:26.170022+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:15:26.170022+0000","last_undegraded":"2023-10-14T01:15:26.170022+0000","last_fullsized":"2023-10-14T01:15:26.170022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'465799","reported_seq":928188,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:28.316107+0000","last_change":"2023-10-14T01:15:27.364720+0000","last_active":"2023-10-14T01:15:28.316107+0000","last_peered":"2023-10-14T01:15:28.316107+0000","last_clean":"2023-10-14T01:15:28.316107+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:15:28.316107+0000","last_undegraded":"2023-10-14T01:15:28.316107+0000","last_fullsized":"2023-10-14T01:15:28.316107+0000","mapping_epoch":15,"log_start":"120'455723","ondisk_log_start":"120'455723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6661820416,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":464531,"num_read_kb":1828160,"num_write":463403,"num_write_kb":1853612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2893,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:24.023423+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:15:24.023423+0000","last_peered":"2023-10-14T01:15:24.023423+0000","last_clean":"2023-10-14T01:15:24.023423+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:15:24.023423+0000","last_undegraded":"2023-10-14T01:15:24.023423+0000","last_fullsized":"2023-10-14T01:15:24.023423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'485203","reported_seq":965642,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:28.766185+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:15:28.766185+0000","last_peered":"2023-10-14T01:15:28.766185+0000","last_clean":"2023-10-14T01:15:28.766185+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:15:28.766185+0000","last_undegraded":"2023-10-14T01:15:28.766185+0000","last_fullsized":"2023-10-14T01:15:28.766185+0000","mapping_epoch":15,"log_start":"120'475128","ondisk_log_start":"120'475128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1269,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 224s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6911152128,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":482583,"num_read_kb":1899588,"num_write":481864,"num_write_kb":1927456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3043,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:15:24.015472+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:15:24.015472+0000","last_peered":"2023-10-14T01:15:24.015472+0000","last_clean":"2023-10-14T01:15:24.015472+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:15:24.015472+0000","last_undegraded":"2023-10-14T01:15:24.015472+0000","last_fullsized":"2023-10-14T01:15:24.015472+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'461166","reported_seq":918003,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:28.316648+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:15:28.316648+0000","last_peered":"2023-10-14T01:15:28.316648+0000","last_clean":"2023-10-14T01:15:28.316648+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:15:28.316648+0000","last_undegraded":"2023-10-14T01:15:28.316648+0000","last_fullsized":"2023-10-14T01:15:28.316648+0000","mapping_epoch":15,"log_start":"120'451125","ondisk_log_start":"120'451125","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 202s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6570733587,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":458861,"num_read_kb":1805416,"num_write":457894,"num_write_kb":1831570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'9348","reported_seq":13517,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.979960+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:15:27.979960+0000","last_peered":"2023-10-14T01:15:27.979960+0000","last_clean":"2023-10-14T01:15:27.979960+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:15:27.979960+0000","last_undegraded":"2023-10-14T01:15:27.979960+0000","last_fullsized":"2023-10-14T01:15:27.979960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":294,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-14T01:14:16.224358+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:14:16.224358+0000","last_peered":"2023-10-14T01:14:16.224358+0000","last_clean":"2023-10-14T01:14:16.224358+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:14:16.224358+0000","last_undegraded":"2023-10-14T01:14:16.224358+0000","last_fullsized":"2023-10-14T01:14:16.224358+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27158405139,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1894481,"num_read_kb":7456252,"num_write":1891187,"num_write_kb":7564742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15491366912,"data_stored":24673730598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":44957,"ondisk_log_size":44957,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739215,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9077076,"kb_used_data":7564364,"kb_used_omap":1043,"kb_used_meta":1511660,"kb_avail":85294764,"statfs":{"total":96636764160,"available":87341838336,"internally_reserved":0,"allocated":7745908736,"data_stored":12336291951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1547940087},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:29 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.443},"min":{"1min":0.217,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.755,"5min":2.198,"15min":27.402000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.46200000000000002,"15min":0.54300000000000004},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.183},"max":{"1min":3.7349999999999999,"5min":5.4050000000000002,"15min":27.538},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:14:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.47499999999999998,"15min":0.42499999999999999},"min":{"1min":0.23000000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.55700000000000005,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.46600000000000003,"15min":0.45400000000000001},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":0.626,"5min":4.1699999999999999,"15min":12.361000000000001},"last":0.29799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739214,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5253504,"kb_used_data":3686516,"kb_used_omap":472,"kb_used_meta":1566503,"kb_avail":89118336,"statfs":{"total":96636764160,"available":91257176064,"internally_reserved":0,"allocated":3774992384,"data_stored":6015450355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1604099791},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.495,"15min":0.439},"min":{"1min":0.214,"5min":0.182,"15min":0.16900000000000001},"max":{"1min":2.738,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.53600000000000003},"min":{"1min":0.246,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":2.6379999999999999,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.38}]},{"osd":2,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41599999999999998,"15min":0.50900000000000001},"min":{"1min":0.214,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":0.48499999999999999,"5min":4.1289999999999996,"15min":15.891},"last":0.216},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.53700000000000003,"15min":0.56200000000000006},"min":{"1min":0.254,"5min":0.221,"15min":0.21299999999999999},"max":{"1min":2.6720000000000002,"5min":5.835,"15min":7.1600000000000001},"last":0.41399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739213,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5827608,"kb_used_data":3881036,"kb_used_omap":500,"kb_used_meta":1946059,"kb_avail":88544232,"statfs":{"total":96636764160,"available":90669293568,"internally_reserved":0,"allocated":3974180864,"data_stored":6323041424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1992765029},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,1,8,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.47799999999999998},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":4.1539999999999999,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.47199999999999998},"min":{"1min":0.20999999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.48899999999999999,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:14:36 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51500000000000001,"15min":0.49099999999999999},"min":{"1min":0.20100000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.0299999999999998,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":2.7549999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":1.1259999999999999,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3972681728,"data_stored":6322409472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3773628416,"data_stored":6015082515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7745056768,"data_stored":12336238611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:15:29.536 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:29.537 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:29.537 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:29.537 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:29.538 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:37.475 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:15:37.476 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:15:37.476 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:15:37.476 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:15:37.477 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:15:37.477 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:15:37.477 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:15:37.477 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:15:37.478 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:15:37.471+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:15:49.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:50.148 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:15:50.149 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:15:50.178 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2240,"stamp":"2023-10-14T01:15:49.459979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27188485861,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1935367,"num_read_kb":7513538,"num_write":1899521,"num_write_kb":7597143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85363,"ondisk_log_size":85363,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20390292,"kb_used_data":15191932,"kb_used_omap":2016,"kb_used_meta":5196319,"kb_avail":262725228,"statfs":{"total":289910292480,"available":269030633472,"internally_reserved":0,"allocated":15556538368,"data_stored":24736240114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5321031211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,8,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22175744,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5371,"num_read_kb":21436,"num_write":5667,"num_write_kb":22668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001778"},"pg_stats":[{"pgid":"2.7","version":"122'6343","reported_seq":9401,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.160790+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:15:48.160790+0000","last_peered":"2023-10-14T01:15:48.160790+0000","last_clean":"2023-10-14T01:15:48.160790+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:15:48.160790+0000","last_undegraded":"2023-10-14T01:15:48.160790+0000","last_fullsized":"2023-10-14T01:15:48.160790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'9439","reported_seq":15204,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.164484+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:15:48.164484+0000","last_peered":"2023-10-14T01:15:48.164484+0000","last_clean":"2023-10-14T01:15:48.164484+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:15:48.164484+0000","last_undegraded":"2023-10-14T01:15:48.164484+0000","last_fullsized":"2023-10-14T01:15:48.164484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'6316","reported_seq":9173,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.170881+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:15:48.170881+0000","last_peered":"2023-10-14T01:15:48.170881+0000","last_clean":"2023-10-14T01:15:48.170881+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:15:48.170881+0000","last_undegraded":"2023-10-14T01:15:48.170881+0000","last_fullsized":"2023-10-14T01:15:48.170881+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'15564","reported_seq":23176,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.192161+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:15:48.192161+0000","last_peered":"2023-10-14T01:15:48.192161+0000","last_clean":"2023-10-14T01:15:48.192161+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:15:48.192161+0000","last_undegraded":"2023-10-14T01:15:48.192161+0000","last_fullsized":"2023-10-14T01:15:48.192161+0000","mapping_epoch":12,"log_start":"83'5500","ondisk_log_start":"83'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'493439","reported_seq":981602,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:48.897678+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:15:48.897678+0000","last_peered":"2023-10-14T01:15:48.897678+0000","last_clean":"2023-10-14T01:15:48.897678+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:15:48.897678+0000","last_undegraded":"2023-10-14T01:15:48.897678+0000","last_fullsized":"2023-10-14T01:15:48.897678+0000","mapping_epoch":15,"log_start":"120'483434","ondisk_log_start":"120'483434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1282,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 243s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7022579712,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":490356,"num_read_kb":1930472,"num_write":490039,"num_write_kb":1960156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'3290","reported_seq":4901,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:42.743948+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:15:42.743948+0000","last_peered":"2023-10-14T01:15:42.743948+0000","last_clean":"2023-10-14T01:15:42.743948+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:15:42.743948+0000","last_undegraded":"2023-10-14T01:15:42.743948+0000","last_fullsized":"2023-10-14T01:15:42.743948+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3290,"log_dups_size":0,"ondisk_log_size":3290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'467637","reported_seq":931772,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.393845+0000","last_change":"2023-10-14T01:15:46.515904+0000","last_active":"2023-10-14T01:15:48.393845+0000","last_peered":"2023-10-14T01:15:48.393845+0000","last_clean":"2023-10-14T01:15:48.393845+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:15:48.393845+0000","last_undegraded":"2023-10-14T01:15:48.393845+0000","last_fullsized":"2023-10-14T01:15:48.393845+0000","mapping_epoch":15,"log_start":"120'457623","ondisk_log_start":"120'457623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6669053952,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":466267,"num_read_kb":1835088,"num_write":465241,"num_write_kb":1860964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2901,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:44.495123+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:15:44.495123+0000","last_peered":"2023-10-14T01:15:44.495123+0000","last_clean":"2023-10-14T01:15:44.495123+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:15:44.495123+0000","last_undegraded":"2023-10-14T01:15:44.495123+0000","last_fullsized":"2023-10-14T01:15:44.495123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'487126","reported_seq":969411,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:48.885135+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:15:48.885135+0000","last_peered":"2023-10-14T01:15:48.885135+0000","last_clean":"2023-10-14T01:15:48.885135+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:15:48.885135+0000","last_undegraded":"2023-10-14T01:15:48.885135+0000","last_fullsized":"2023-10-14T01:15:48.885135+0000","mapping_epoch":15,"log_start":"120'477028","ondisk_log_start":"120'477028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1384,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 244s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6918299648,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":484412,"num_read_kb":1906884,"num_write":483704,"num_write_kb":1934816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3052,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:44.495268+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:15:44.495268+0000","last_peered":"2023-10-14T01:15:44.495268+0000","last_clean":"2023-10-14T01:15:44.495268+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:15:44.495268+0000","last_undegraded":"2023-10-14T01:15:44.495268+0000","last_fullsized":"2023-10-14T01:15:44.495268+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'463109","reported_seq":921680,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:48.400976+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:15:48.400976+0000","last_peered":"2023-10-14T01:15:48.400976+0000","last_clean":"2023-10-14T01:15:48.400976+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:15:48.400976+0000","last_undegraded":"2023-10-14T01:15:48.400976+0000","last_fullsized":"2023-10-14T01:15:48.400976+0000","mapping_epoch":15,"log_start":"120'453025","ondisk_log_start":"120'453025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1491,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 222s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6577958931,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":460584,"num_read_kb":1812288,"num_write":459741,"num_write_kb":1838958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'9387","reported_seq":13568,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.182592+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:15:48.182592+0000","last_peered":"2023-10-14T01:15:48.182592+0000","last_clean":"2023-10-14T01:15:48.182592+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:15:48.182592+0000","last_undegraded":"2023-10-14T01:15:48.182592+0000","last_fullsized":"2023-10-14T01:15:48.182592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9387,"log_dups_size":0,"ondisk_log_size":9387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":295,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:26.232536+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:15:26.232536+0000","last_peered":"2023-10-14T01:15:26.232536+0000","last_clean":"2023-10-14T01:15:26.232536+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:15:26.232536+0000","last_undegraded":"2023-10-14T01:15:26.232536+0000","last_fullsized":"2023-10-14T01:15:26.232536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27187892243,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1901619,"num_read_kb":7484732,"num_write":1898725,"num_write_kb":7594894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15553130496,"data_stored":24735494182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40201,"ondisk_log_size":40201,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45063,"ondisk_log_size":45063,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739219,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9193084,"kb_used_data":7594292,"kb_used_omap":1043,"kb_used_meta":1597740,"kb_avail":85178756,"statfs":{"total":96636764160,"available":87223046144,"internally_reserved":0,"allocated":7776555008,"data_stored":12366938223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1636086007},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.32700000000000001,"15min":0.443},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.46300000000000002,"5min":2.198,"15min":27.402000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.45900000000000002,"15min":0.54300000000000004},"min":{"1min":0.23000000000000001,"5min":0.191,"15min":0.183},"max":{"1min":0.498,"5min":5.4050000000000002,"15min":27.538},"last":0.311}]},{"osd":1,"last update":"Sat Oct 14 01:14:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.47499999999999998,"15min":0.42499999999999999},"min":{"1min":0.23000000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.55700000000000005,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.46600000000000003,"15min":0.45400000000000001},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":0.626,"5min":4.1699999999999999,"15min":12.361000000000001},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739218,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5300160,"kb_used_data":3701304,"kb_used_omap":472,"kb_used_meta":1598375,"kb_avail":89071680,"statfs":{"total":96636764160,"available":91209400320,"internally_reserved":0,"allocated":3790135296,"data_stored":6030593267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1636736719},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.55100000000000005,"15min":0.45800000000000002},"min":{"1min":0.216,"5min":0.184,"15min":0.16900000000000001},"max":{"1min":4.1180000000000003,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.63500000000000001,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.26100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.39500000000000002,"15min":0.497},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.182},"max":{"1min":0.49299999999999999,"5min":4.1289999999999996,"15min":15.891},"last":1.9390000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.53400000000000003,"15min":0.54100000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59499999999999997,"5min":5.835,"15min":7.1600000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739217,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5897048,"kb_used_data":3896336,"kb_used_omap":500,"kb_used_meta":2000203,"kb_avail":88474792,"statfs":{"total":96636764160,"available":90598187008,"internally_reserved":0,"allocated":3989848064,"data_stored":6338708624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":2048208485},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.45200000000000001,"15min":0.46899999999999997},"min":{"1min":0.185,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":1.0149999999999999,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.48299999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.5739999999999998,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":3.3109999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.60099999999999998,"15min":0.49199999999999999},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.915,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":3.1539999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":4.8049999999999997,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3988455424,"data_stored":6338183168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3788726272,"data_stored":6030180371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7775948800,"data_stored":12367130643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:15:50.180 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:50.181 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:50.181 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:50.181 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:50.182 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:58.967 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:15:58.963+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:16:10.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:10.710 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:16:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:16:10.739 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2250,"stamp":"2023-10-14T01:16:09.467347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27218915045,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1942661,"num_read_kb":7542654,"num_write":1907312,"num_write_kb":7628307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85412,"ondisk_log_size":85412,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20653624,"kb_used_data":15254432,"kb_used_omap":2016,"kb_used_meta":5397151,"kb_avail":262461896,"statfs":{"total":289910292480,"available":268760981504,"internally_reserved":0,"allocated":15620538368,"data_stored":24800240114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5526683179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,45,18,12,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22962176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5557,"num_read_kb":22188,"num_write":5862,"num_write_kb":23448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001991"},"pg_stats":[{"pgid":"2.7","version":"122'6365","reported_seq":9431,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:07.006648+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:16:07.006648+0000","last_peered":"2023-10-14T01:16:07.006648+0000","last_clean":"2023-10-14T01:16:07.006648+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:16:07.006648+0000","last_undegraded":"2023-10-14T01:16:07.006648+0000","last_fullsized":"2023-10-14T01:16:07.006648+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'9472","reported_seq":15249,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:07.020281+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:16:07.020281+0000","last_peered":"2023-10-14T01:16:07.020281+0000","last_clean":"2023-10-14T01:16:07.020281+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:16:07.020281+0000","last_undegraded":"2023-10-14T01:16:07.020281+0000","last_fullsized":"2023-10-14T01:16:07.020281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'6338","reported_seq":9203,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:07.017076+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:16:07.017076+0000","last_peered":"2023-10-14T01:16:07.017076+0000","last_clean":"2023-10-14T01:16:07.017076+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:16:07.017076+0000","last_undegraded":"2023-10-14T01:16:07.017076+0000","last_fullsized":"2023-10-14T01:16:07.017076+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'15619","reported_seq":23251,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:07.030389+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:16:07.030389+0000","last_peered":"2023-10-14T01:16:07.030389+0000","last_clean":"2023-10-14T01:16:07.030389+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:16:07.030389+0000","last_undegraded":"2023-10-14T01:16:07.030389+0000","last_fullsized":"2023-10-14T01:16:07.030389+0000","mapping_epoch":12,"log_start":"83'5600","ondisk_log_start":"83'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'495468","reported_seq":985526,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:09.057337+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:16:09.057337+0000","last_peered":"2023-10-14T01:16:09.057337+0000","last_clean":"2023-10-14T01:16:09.057337+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:16:09.057337+0000","last_undegraded":"2023-10-14T01:16:09.057337+0000","last_fullsized":"2023-10-14T01:16:09.057337+0000","mapping_epoch":15,"log_start":"121'485434","ondisk_log_start":"121'485434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1330,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 263s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7030394880,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":492243,"num_read_kb":1937992,"num_write":492034,"num_write_kb":1968136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'3301","reported_seq":4916,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:06.234796+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:16:06.234796+0000","last_peered":"2023-10-14T01:16:06.234796+0000","last_clean":"2023-10-14T01:16:06.234796+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:16:06.234796+0000","last_undegraded":"2023-10-14T01:16:06.234796+0000","last_fullsized":"2023-10-14T01:16:06.234796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3301,"log_dups_size":0,"ondisk_log_size":3301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'469572","reported_seq":935525,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:08.539628+0000","last_change":"2023-10-14T01:16:04.552174+0000","last_active":"2023-10-14T01:16:08.539628+0000","last_peered":"2023-10-14T01:16:08.539628+0000","last_clean":"2023-10-14T01:16:08.539628+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:16:08.539628+0000","last_undegraded":"2023-10-14T01:16:08.539628+0000","last_fullsized":"2023-10-14T01:16:08.539628+0000","mapping_epoch":15,"log_start":"121'459523","ondisk_log_start":"121'459523","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6676623360,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":468074,"num_read_kb":1842312,"num_write":467176,"num_write_kb":1868704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2910,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:05.007547+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:16:05.007547+0000","last_peered":"2023-10-14T01:16:05.007547+0000","last_clean":"2023-10-14T01:16:05.007547+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:16:05.007547+0000","last_undegraded":"2023-10-14T01:16:05.007547+0000","last_fullsized":"2023-10-14T01:16:05.007547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'489191","reported_seq":973306,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:09.047821+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:16:09.047821+0000","last_peered":"2023-10-14T01:16:09.047821+0000","last_clean":"2023-10-14T01:16:09.047821+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:16:09.047821+0000","last_undegraded":"2023-10-14T01:16:09.047821+0000","last_fullsized":"2023-10-14T01:16:09.047821+0000","mapping_epoch":15,"log_start":"121'479128","ondisk_log_start":"121'479128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1484,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 264s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6926016512,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":486232,"num_read_kb":1914144,"num_write":485698,"num_write_kb":1942792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3060,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:04.986760+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:16:04.986760+0000","last_peered":"2023-10-14T01:16:04.986760+0000","last_clean":"2023-10-14T01:16:04.986760+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:16:04.986760+0000","last_undegraded":"2023-10-14T01:16:04.986760+0000","last_fullsized":"2023-10-14T01:16:04.986760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'465053","reported_seq":925406,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:08.574711+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:16:08.574711+0000","last_peered":"2023-10-14T01:16:08.574711+0000","last_clean":"2023-10-14T01:16:08.574711+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:16:08.574711+0000","last_undegraded":"2023-10-14T01:16:08.574711+0000","last_fullsized":"2023-10-14T01:16:08.574711+0000","mapping_epoch":15,"log_start":"121'455025","ondisk_log_start":"121'455025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1599,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 243s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6585286675,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":462364,"num_read_kb":1819400,"num_write":461608,"num_write_kb":1846426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'9420","reported_seq":13613,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:07.102920+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:16:07.102920+0000","last_peered":"2023-10-14T01:16:07.102920+0000","last_clean":"2023-10-14T01:16:07.102920+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:16:07.102920+0000","last_undegraded":"2023-10-14T01:16:07.102920+0000","last_fullsized":"2023-10-14T01:16:07.102920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9420,"log_dups_size":0,"ondisk_log_size":9420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":295,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:26.232536+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:15:26.232536+0000","last_peered":"2023-10-14T01:15:26.232536+0000","last_clean":"2023-10-14T01:15:26.232536+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:15:26.232536+0000","last_undegraded":"2023-10-14T01:15:26.232536+0000","last_fullsized":"2023-10-14T01:15:26.232536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27218321427,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1908913,"num_read_kb":7513848,"num_write":1906516,"num_write_kb":7626058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15616884736,"data_stored":24799248422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40174,"ondisk_log_size":40174,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45139,"ondisk_log_size":45139,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739223,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9319892,"kb_used_data":7625164,"kb_used_omap":1043,"kb_used_meta":1693676,"kb_avail":85051948,"statfs":{"total":96636764160,"available":87093194752,"internally_reserved":0,"allocated":7808167936,"data_stored":12398551151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1734324471},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.32700000000000001,"15min":0.443},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.46300000000000002,"5min":2.198,"15min":27.402000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.45900000000000002,"15min":0.54300000000000004},"min":{"1min":0.23000000000000001,"5min":0.191,"15min":0.183},"max":{"1min":0.498,"5min":5.4050000000000002,"15min":27.538},"last":0.245}]},{"osd":1,"last update":"Sat Oct 14 01:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.45600000000000002,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.0620000000000001,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44,"15min":0.41999999999999998},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":1.1080000000000001,"5min":4.1699999999999999,"15min":4.2320000000000002},"last":0.32500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739222,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5380572,"kb_used_data":3717076,"kb_used_omap":472,"kb_used_meta":1663015,"kb_avail":88991268,"statfs":{"total":96636764160,"available":91127058432,"internally_reserved":0,"allocated":3806285824,"data_stored":6046743795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1702928079},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,3,10,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.55100000000000005,"15min":0.45800000000000002},"min":{"1min":0.216,"5min":0.184,"15min":0.16900000000000001},"max":{"1min":4.1180000000000003,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.184},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.63500000000000001,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.39500000000000002,"15min":0.497},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.182},"max":{"1min":0.49299999999999999,"5min":4.1289999999999996,"15min":15.891},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.53400000000000003,"15min":0.54100000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59499999999999997,"5min":5.835,"15min":7.1600000000000001},"last":0.39900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739221,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5953160,"kb_used_data":3912192,"kb_used_omap":500,"kb_used_meta":2040459,"kb_avail":88418680,"statfs":{"total":96636764160,"available":90540728320,"internally_reserved":0,"allocated":4006084608,"data_stored":6354945168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":2089430629},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,41,6,4,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.45200000000000001,"15min":0.46899999999999997},"min":{"1min":0.185,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":1.0149999999999999,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.48299999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.5739999999999998,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.60099999999999998,"15min":0.49199999999999999},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.915,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.309},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":4.8049999999999997,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4004802560,"data_stored":6354530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3804295168,"data_stored":6045749267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7807787008,"data_stored":12398968851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:16:10.740 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:10.741 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:10.741 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:10.742 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:10.742 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:27.680 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:16:27.674+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:16:27.681 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:16:27.674+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:16:27.681 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:16:27.674+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:16:27.699 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:16:27.694+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:16:30.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:31.249 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:16:31.250 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:16:31.287 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2260,"stamp":"2023-10-14T01:16:29.470702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27248471781,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1949982,"num_read_kb":7571882,"num_write":1914883,"num_write_kb":7658591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85692,"ondisk_log_size":85692,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20361092,"kb_used_data":15314732,"kb_used_omap":2016,"kb_used_meta":5044319,"kb_avail":262754428,"statfs":{"total":289910292480,"available":269060534272,"internally_reserved":0,"allocated":15682285568,"data_stored":24861987314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5165383211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,3,16,7,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21856256,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5449,"num_read_kb":21756,"num_write":5592,"num_write_kb":22368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001975"},"pg_stats":[{"pgid":"2.7","version":"122'6390","reported_seq":9464,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:28.596617+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:16:28.596617+0000","last_peered":"2023-10-14T01:16:28.596617+0000","last_clean":"2023-10-14T01:16:28.596617+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:16:28.596617+0000","last_undegraded":"2023-10-14T01:16:28.596617+0000","last_fullsized":"2023-10-14T01:16:28.596617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'9508","reported_seq":15297,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:28.620140+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:16:28.620140+0000","last_peered":"2023-10-14T01:16:28.620140+0000","last_clean":"2023-10-14T01:16:28.620140+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:16:28.620140+0000","last_undegraded":"2023-10-14T01:16:28.620140+0000","last_fullsized":"2023-10-14T01:16:28.620140+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9508,"log_dups_size":0,"ondisk_log_size":9508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'6363","reported_seq":9236,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:28.595472+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:16:28.595472+0000","last_peered":"2023-10-14T01:16:28.595472+0000","last_clean":"2023-10-14T01:16:28.595472+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:16:28.595472+0000","last_undegraded":"2023-10-14T01:16:28.595472+0000","last_fullsized":"2023-10-14T01:16:28.595472+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'15680","reported_seq":23332,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:28.637921+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:16:28.637921+0000","last_peered":"2023-10-14T01:16:28.637921+0000","last_clean":"2023-10-14T01:16:28.637921+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:16:28.637921+0000","last_undegraded":"2023-10-14T01:16:28.637921+0000","last_fullsized":"2023-10-14T01:16:28.637921+0000","mapping_epoch":12,"log_start":"83'5600","ondisk_log_start":"83'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'497494","reported_seq":989470,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:29.174848+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:16:29.174848+0000","last_peered":"2023-10-14T01:16:29.174848+0000","last_clean":"2023-10-14T01:16:29.174848+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:16:29.174848+0000","last_undegraded":"2023-10-14T01:16:29.174848+0000","last_fullsized":"2023-10-14T01:16:29.174848+0000","mapping_epoch":15,"log_start":"121'487434","ondisk_log_start":"121'487434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1461,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 284s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7037972480,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":494147,"num_read_kb":1945584,"num_write":493970,"num_write_kb":1975880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'3312","reported_seq":4931,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:27.434739+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:16:27.434739+0000","last_peered":"2023-10-14T01:16:27.434739+0000","last_clean":"2023-10-14T01:16:27.434739+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:16:27.434739+0000","last_undegraded":"2023-10-14T01:16:27.434739+0000","last_fullsized":"2023-10-14T01:16:27.434739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3312,"log_dups_size":0,"ondisk_log_size":3312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'471405","reported_seq":939123,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:28.655667+0000","last_change":"2023-10-14T01:16:24.698249+0000","last_active":"2023-10-14T01:16:28.655667+0000","last_peered":"2023-10-14T01:16:28.655667+0000","last_clean":"2023-10-14T01:16:28.655667+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:16:28.655667+0000","last_undegraded":"2023-10-14T01:16:28.655667+0000","last_fullsized":"2023-10-14T01:16:28.655667+0000","mapping_epoch":15,"log_start":"121'461323","ondisk_log_start":"121'461323","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6683869184,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":469835,"num_read_kb":1849344,"num_write":469009,"num_write_kb":1876036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2918,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:25.476971+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:16:25.476971+0000","last_peered":"2023-10-14T01:16:25.476971+0000","last_clean":"2023-10-14T01:16:25.476971+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:16:25.476971+0000","last_undegraded":"2023-10-14T01:16:25.476971+0000","last_fullsized":"2023-10-14T01:16:25.476971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'491257","reported_seq":977200,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:29.148766+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:16:29.148766+0000","last_peered":"2023-10-14T01:16:29.148766+0000","last_clean":"2023-10-14T01:16:29.148766+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:16:29.148766+0000","last_undegraded":"2023-10-14T01:16:29.148766+0000","last_fullsized":"2023-10-14T01:16:29.148766+0000","mapping_epoch":15,"log_start":"121'481228","ondisk_log_start":"121'481228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1573,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 285s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6933815296,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":488052,"num_read_kb":1921420,"num_write":487705,"num_write_kb":1950820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3068,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:25.455165+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:16:25.455165+0000","last_peered":"2023-10-14T01:16:25.455165+0000","last_clean":"2023-10-14T01:16:25.455165+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:16:25.455165+0000","last_undegraded":"2023-10-14T01:16:25.455165+0000","last_fullsized":"2023-10-14T01:16:25.455165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'466913","reported_seq":929124,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:28.624949+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:16:28.624949+0000","last_peered":"2023-10-14T01:16:28.624949+0000","last_clean":"2023-10-14T01:16:28.624949+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:16:28.624949+0000","last_undegraded":"2023-10-14T01:16:28.624949+0000","last_fullsized":"2023-10-14T01:16:28.624949+0000","mapping_epoch":15,"log_start":"121'456825","ondisk_log_start":"121'456825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1696,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 263s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6592221203,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":464200,"num_read_kb":1826728,"num_write":463403,"num_write_kb":1853606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'9457","reported_seq":13662,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:16:28.622149+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:16:28.622149+0000","last_peered":"2023-10-14T01:16:28.622149+0000","last_clean":"2023-10-14T01:16:28.622149+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:16:28.622149+0000","last_undegraded":"2023-10-14T01:16:28.622149+0000","last_fullsized":"2023-10-14T01:16:28.622149+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9457,"log_dups_size":0,"ondisk_log_size":9457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":295,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:26.232536+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:15:26.232536+0000","last_peered":"2023-10-14T01:15:26.232536+0000","last_clean":"2023-10-14T01:15:26.232536+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:15:26.232536+0000","last_undegraded":"2023-10-14T01:15:26.232536+0000","last_fullsized":"2023-10-14T01:15:26.232536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27247878163,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1916234,"num_read_kb":7543076,"num_write":1914087,"num_write_kb":7656342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15679041536,"data_stored":24861405222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45334,"ondisk_log_size":45334,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739227,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9452936,"kb_used_data":7655552,"kb_used_omap":1043,"kb_used_meta":1796332,"kb_avail":84918904,"statfs":{"total":96636764160,"available":86956957696,"internally_reserved":0,"allocated":7839285248,"data_stored":12429668463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1839444215},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.32700000000000001,"15min":0.443},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.46300000000000002,"5min":2.198,"15min":27.402000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.45900000000000002,"15min":0.54300000000000004},"min":{"1min":0.23000000000000001,"5min":0.191,"15min":0.183},"max":{"1min":0.498,"5min":5.4050000000000002,"15min":27.538},"last":0.316}]},{"osd":1,"last update":"Sat Oct 14 01:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.45600000000000002,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.0620000000000001,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44,"15min":0.41999999999999998},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":1.1080000000000001,"5min":4.1699999999999999,"15min":4.2320000000000002},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359739226,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5426376,"kb_used_data":3731008,"kb_used_omap":472,"kb_used_meta":1694887,"kb_avail":88945464,"statfs":{"total":96636764160,"available":91080155136,"internally_reserved":0,"allocated":3820552192,"data_stored":6061010163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1735565007},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.55100000000000005,"15min":0.45800000000000002},"min":{"1min":0.216,"5min":0.184,"15min":0.16900000000000001},"max":{"1min":4.1180000000000003,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.247},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.63500000000000001,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.39500000000000002,"15min":0.497},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.182},"max":{"1min":0.49299999999999999,"5min":4.1289999999999996,"15min":15.891},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.53400000000000003,"15min":0.54100000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":0.59499999999999997,"5min":5.835,"15min":7.1600000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359739225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5481780,"kb_used_data":3928172,"kb_used_omap":500,"kb_used_meta":1553099,"kb_avail":88890060,"statfs":{"total":96636764160,"available":91023421440,"internally_reserved":0,"allocated":4022448128,"data_stored":6371308688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1590373989},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,8,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.45200000000000001,"15min":0.46899999999999997},"min":{"1min":0.185,"5min":0.17599999999999999,"15min":0.156},"max":{"1min":1.0149999999999999,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.48299999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.5739999999999998,"5min":5.0060000000000002,"15min":6.7069999999999999},"last":0.20899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.60099999999999998,"15min":0.49199999999999999},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.915,"5min":5.1909999999999998,"15min":7.4690000000000003},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":4.8049999999999997,"5min":4.9690000000000003,"15min":18.966999999999999},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4020940800,"data_stored":6370668544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3819163648,"data_stored":6060617747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7838937088,"data_stored":12430118931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:16:31.289 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:31.289 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:31.290 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:31.290 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:31.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:16:31.775 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:16:31.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:16:32.332 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:16:32.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:16:32.893 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:16:32.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:16:33.502 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:16:33.535 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:53.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:54.028 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:16:54.029 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:16:54.054 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2273,"stamp":"2023-10-14T01:16:53.474709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27277631205,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1957301,"num_read_kb":7601094,"num_write":1922349,"num_write_kb":7688455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85736,"ondisk_log_size":85736,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20624516,"kb_used_data":15382316,"kb_used_omap":2016,"kb_used_meta":5240159,"kb_avail":262491004,"statfs":{"total":289910292480,"available":268790788096,"internally_reserved":0,"allocated":15751491584,"data_stored":24931170161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5365923371},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,0,3,14,22,24,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14921728,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3747,"num_read_kb":14956,"num_write":3812,"num_write_kb":15248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001949"},"pg_stats":[{"pgid":"2.7","version":"123'6410","reported_seq":9493,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.487653+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:16:48.487653+0000","last_peered":"2023-10-14T01:16:48.487653+0000","last_clean":"2023-10-14T01:16:48.487653+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:16:48.487653+0000","last_undegraded":"2023-10-14T01:16:48.487653+0000","last_fullsized":"2023-10-14T01:16:48.487653+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"123'9538","reported_seq":15340,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.495966+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:16:48.495966+0000","last_peered":"2023-10-14T01:16:48.495966+0000","last_clean":"2023-10-14T01:16:48.495966+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:16:48.495966+0000","last_undegraded":"2023-10-14T01:16:48.495966+0000","last_fullsized":"2023-10-14T01:16:48.495966+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"123'6383","reported_seq":9266,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.480628+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:16:48.480628+0000","last_peered":"2023-10-14T01:16:48.480628+0000","last_clean":"2023-10-14T01:16:48.480628+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:16:48.480628+0000","last_undegraded":"2023-10-14T01:16:48.480628+0000","last_fullsized":"2023-10-14T01:16:48.480628+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"123'15730","reported_seq":23404,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.506071+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:16:48.506071+0000","last_peered":"2023-10-14T01:16:48.506071+0000","last_clean":"2023-10-14T01:16:48.506071+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:16:48.506071+0000","last_undegraded":"2023-10-14T01:16:48.506071+0000","last_fullsized":"2023-10-14T01:16:48.506071+0000","mapping_epoch":12,"log_start":"85'5700","ondisk_log_start":"85'5700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"123'499482","reported_seq":993349,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:49.266403+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:16:49.266403+0000","last_peered":"2023-10-14T01:16:49.266403+0000","last_clean":"2023-10-14T01:16:49.266403+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:16:49.266403+0000","last_undegraded":"2023-10-14T01:16:49.266403+0000","last_fullsized":"2023-10-14T01:16:49.266403+0000","mapping_epoch":15,"log_start":"121'489435","ondisk_log_start":"121'489435","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1561,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 304s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7045472256,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":496023,"num_read_kb":1953068,"num_write":495888,"num_write_kb":1983552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"123'3319","reported_seq":4944,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.299614+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:16:48.299614+0000","last_peered":"2023-10-14T01:16:48.299614+0000","last_clean":"2023-10-14T01:16:48.299614+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:16:48.299614+0000","last_undegraded":"2023-10-14T01:16:48.299614+0000","last_fullsized":"2023-10-14T01:16:48.299614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3319,"log_dups_size":0,"ondisk_log_size":3319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"123'473300","reported_seq":942863,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.725574+0000","last_change":"2023-10-14T01:16:48.018895+0000","last_active":"2023-10-14T01:16:48.725574+0000","last_peered":"2023-10-14T01:16:48.725574+0000","last_clean":"2023-10-14T01:16:48.725574+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:16:48.725574+0000","last_undegraded":"2023-10-14T01:16:48.725574+0000","last_fullsized":"2023-10-14T01:16:48.725574+0000","mapping_epoch":15,"log_start":"121'463223","ondisk_log_start":"121'463223","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6691323904,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":471670,"num_read_kb":1856676,"num_write":470904,"num_write_kb":1883616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2929,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:51.247154+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:16:51.247154+0000","last_peered":"2023-10-14T01:16:51.247154+0000","last_clean":"2023-10-14T01:16:51.247154+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:16:51.247154+0000","last_undegraded":"2023-10-14T01:16:51.247154+0000","last_fullsized":"2023-10-14T01:16:51.247154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"123'493211","reported_seq":981074,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:49.253301+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:16:49.253301+0000","last_peered":"2023-10-14T01:16:49.253301+0000","last_clean":"2023-10-14T01:16:49.253301+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:16:49.253301+0000","last_undegraded":"2023-10-14T01:16:49.253301+0000","last_fullsized":"2023-10-14T01:16:49.253301+0000","mapping_epoch":15,"log_start":"121'483128","ondisk_log_start":"121'483128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1684,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 305s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6941069312,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":489956,"num_read_kb":1929024,"num_write":489582,"num_write_kb":1958328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3079,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:51.055253+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:16:51.055253+0000","last_peered":"2023-10-14T01:16:51.055253+0000","last_clean":"2023-10-14T01:16:51.055253+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:16:51.055253+0000","last_undegraded":"2023-10-14T01:16:51.055253+0000","last_fullsized":"2023-10-14T01:16:51.055253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"123'468764","reported_seq":932692,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:48.730504+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:16:48.730504+0000","last_peered":"2023-10-14T01:16:48.730504+0000","last_clean":"2023-10-14T01:16:48.730504+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:16:48.730504+0000","last_undegraded":"2023-10-14T01:16:48.730504+0000","last_fullsized":"2023-10-14T01:16:48.730504+0000","mapping_epoch":15,"log_start":"121'458725","ondisk_log_start":"121'458725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1807,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 283s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6599172115,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":465904,"num_read_kb":1833520,"num_write":465179,"num_write_kb":1860710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"123'9487","reported_seq":13706,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:16:48.531954+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:16:48.531954+0000","last_peered":"2023-10-14T01:16:48.531954+0000","last_clean":"2023-10-14T01:16:48.531954+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:16:48.531954+0000","last_undegraded":"2023-10-14T01:16:48.531954+0000","last_fullsized":"2023-10-14T01:16:48.531954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":296,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:38.355507+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:15:38.355507+0000","last_peered":"2023-10-14T01:15:38.355507+0000","last_clean":"2023-10-14T01:15:38.355507+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:15:38.355507+0000","last_undegraded":"2023-10-14T01:15:38.355507+0000","last_fullsized":"2023-10-14T01:15:38.355507+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27277037587,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1923553,"num_read_kb":7572288,"num_write":1921553,"num_write_kb":7686206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15747858432,"data_stored":24930222118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40246,"ondisk_log_size":40246,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45391,"ondisk_log_size":45391,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739232,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9614096,"kb_used_data":7693064,"kb_used_omap":1043,"kb_used_meta":1919980,"kb_avail":84757744,"statfs":{"total":96636764160,"available":86791929856,"internally_reserved":0,"allocated":7877697536,"data_stored":12468073028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":1966059767},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.34899999999999998,"15min":0.45100000000000001},"min":{"1min":0.19600000000000001,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":4.2560000000000002,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.52500000000000002,"15min":0.56200000000000006},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.183},"max":{"1min":4.1509999999999998,"5min":5.4050000000000002,"15min":27.538},"last":0.311}]},{"osd":1,"last update":"Sat Oct 14 01:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.45600000000000002,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.0620000000000001,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44,"15min":0.41999999999999998},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":1.1080000000000001,"5min":4.1699999999999999,"15min":4.2320000000000002},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5495336,"kb_used_data":3745824,"kb_used_omap":472,"kb_used_meta":1749031,"kb_avail":88876504,"statfs":{"total":96636764160,"available":91009540096,"internally_reserved":0,"allocated":3835723776,"data_stored":6076174024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1791008463},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.57099999999999995,"15min":0.46400000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.16900000000000001},"max":{"1min":2.3780000000000001,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.505},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.222,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.83299999999999996,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Sat Oct 14 01:16:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.47699999999999998,"15min":0.499},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":4.5170000000000003,"5min":4.5170000000000003,"15min":15.891},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.54500000000000004,"15min":0.51100000000000001},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":0.621,"5min":5.835,"15min":5.835},"last":0.35199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5515084,"kb_used_data":3943428,"kb_used_omap":500,"kb_used_meta":1571147,"kb_avail":88856756,"statfs":{"total":96636764160,"available":90989318144,"internally_reserved":0,"allocated":4038070272,"data_stored":6386923109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1608855141},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,5,12,23,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.47399999999999998},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.156},"max":{"1min":1.212,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.45000000000000001},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.16800000000000001},"max":{"1min":3.3109999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54000000000000004,"15min":0.46999999999999997},"min":{"1min":0.21199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.1539999999999999,"5min":3.1629999999999998,"15min":5.1909999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.53500000000000003,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":0.79200000000000004,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4036485120,"data_stored":6386212864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3834195968,"data_stored":6075650067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7877177344,"data_stored":12468359187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:16:54.056 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:54.057 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:54.057 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:54.058 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:54.058 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:14.060 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:14.653 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:17:14.654 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:17:14.687 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2283,"stamp":"2023-10-14T01:17:13.478049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27306487525,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1964847,"num_read_kb":7631198,"num_write":1929786,"num_write_kb":7718203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85907,"ondisk_log_size":85907,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20875404,"kb_used_data":15441716,"kb_used_omap":2016,"kb_used_meta":5431647,"kb_avail":262240116,"statfs":{"total":289910292480,"available":268533878784,"internally_reserved":0,"allocated":15812317184,"data_stored":24991995761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5562007083},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,5,9,15,15,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14450688,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3778,"num_read_kb":15056,"num_write":3732,"num_write_kb":14928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001998"},"pg_stats":[{"pgid":"2.7","version":"123'6438","reported_seq":9530,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:06.415345+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:17:06.415345+0000","last_peered":"2023-10-14T01:17:06.415345+0000","last_clean":"2023-10-14T01:17:06.415345+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:17:06.415345+0000","last_undegraded":"2023-10-14T01:17:06.415345+0000","last_fullsized":"2023-10-14T01:17:06.415345+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"123'9580","reported_seq":15395,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:06.415258+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:17:06.415258+0000","last_peered":"2023-10-14T01:17:06.415258+0000","last_clean":"2023-10-14T01:17:06.415258+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:17:06.415258+0000","last_undegraded":"2023-10-14T01:17:06.415258+0000","last_fullsized":"2023-10-14T01:17:06.415258+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"123'6411","reported_seq":9302,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:06.414951+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:17:06.414951+0000","last_peered":"2023-10-14T01:17:06.414951+0000","last_clean":"2023-10-14T01:17:06.414951+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:17:06.414951+0000","last_undegraded":"2023-10-14T01:17:06.414951+0000","last_fullsized":"2023-10-14T01:17:06.414951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6411,"log_dups_size":0,"ondisk_log_size":6411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"123'15800","reported_seq":23494,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:06.415781+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:17:06.415781+0000","last_peered":"2023-10-14T01:17:06.415781+0000","last_clean":"2023-10-14T01:17:06.415781+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:17:06.415781+0000","last_undegraded":"2023-10-14T01:17:06.415781+0000","last_fullsized":"2023-10-14T01:17:06.415781+0000","mapping_epoch":12,"log_start":"85'5700","ondisk_log_start":"85'5700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"123'501513","reported_seq":997363,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:09.361600+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:17:09.361600+0000","last_peered":"2023-10-14T01:17:09.361600+0000","last_clean":"2023-10-14T01:17:09.361600+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:17:09.361600+0000","last_undegraded":"2023-10-14T01:17:09.361600+0000","last_fullsized":"2023-10-14T01:17:09.361600+0000","mapping_epoch":15,"log_start":"122'491435","ondisk_log_start":"122'491435","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1675,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 324s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7053164544,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":497997,"num_read_kb":1960944,"num_write":497843,"num_write_kb":1991372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"123'3330","reported_seq":4959,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:06.849846+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:17:06.849846+0000","last_peered":"2023-10-14T01:17:06.849846+0000","last_clean":"2023-10-14T01:17:06.849846+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:17:06.849846+0000","last_undegraded":"2023-10-14T01:17:06.849846+0000","last_fullsized":"2023-10-14T01:17:06.849846+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3330,"log_dups_size":0,"ondisk_log_size":3330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"123'475141","reported_seq":946577,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:08.770147+0000","last_change":"2023-10-14T01:17:07.446688+0000","last_active":"2023-10-14T01:17:08.770147+0000","last_peered":"2023-10-14T01:17:08.770147+0000","last_clean":"2023-10-14T01:17:08.770147+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:17:08.770147+0000","last_undegraded":"2023-10-14T01:17:08.770147+0000","last_fullsized":"2023-10-14T01:17:08.770147+0000","mapping_epoch":15,"log_start":"121'465123","ondisk_log_start":"121'465123","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6698459136,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":473543,"num_read_kb":1864152,"num_write":472745,"num_write_kb":1890980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2937,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:11.535451+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:17:11.535451+0000","last_peered":"2023-10-14T01:17:11.535451+0000","last_clean":"2023-10-14T01:17:11.535451+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:17:11.535451+0000","last_undegraded":"2023-10-14T01:17:11.535451+0000","last_fullsized":"2023-10-14T01:17:11.535451+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"123'495182","reported_seq":984943,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:09.355548+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:17:09.355548+0000","last_peered":"2023-10-14T01:17:09.355548+0000","last_clean":"2023-10-14T01:17:09.355548+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:17:09.355548+0000","last_undegraded":"2023-10-14T01:17:09.355548+0000","last_fullsized":"2023-10-14T01:17:09.355548+0000","mapping_epoch":15,"log_start":"122'485128","ondisk_log_start":"122'485128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1794,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 325s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6948315136,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":491845,"num_read_kb":1936552,"num_write":491477,"num_write_kb":1965908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3087,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:11.535077+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:17:11.535077+0000","last_peered":"2023-10-14T01:17:11.535077+0000","last_clean":"2023-10-14T01:17:11.535077+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:17:11.535077+0000","last_undegraded":"2023-10-14T01:17:11.535077+0000","last_fullsized":"2023-10-14T01:17:11.535077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"123'470571","reported_seq":936319,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:08.780371+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:17:08.780371+0000","last_peered":"2023-10-14T01:17:08.780371+0000","last_clean":"2023-10-14T01:17:08.780371+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:17:08.780371+0000","last_undegraded":"2023-10-14T01:17:08.780371+0000","last_fullsized":"2023-10-14T01:17:08.780371+0000","mapping_epoch":15,"log_start":"121'460525","ondisk_log_start":"121'460525","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1896,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6605955091,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":467714,"num_read_kb":1840744,"num_write":466925,"num_write_kb":1867694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"123'9529","reported_seq":13760,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:06.415794+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:17:06.415794+0000","last_peered":"2023-10-14T01:17:06.415794+0000","last_clean":"2023-10-14T01:17:06.415794+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:17:06.415794+0000","last_undegraded":"2023-10-14T01:17:06.415794+0000","last_fullsized":"2023-10-14T01:17:06.415794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":296,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:38.355507+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:15:38.355507+0000","last_peered":"2023-10-14T01:15:38.355507+0000","last_clean":"2023-10-14T01:15:38.355507+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:15:38.355507+0000","last_undegraded":"2023-10-14T01:15:38.355507+0000","last_fullsized":"2023-10-14T01:15:38.355507+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27305893907,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1931099,"num_read_kb":7602392,"num_write":1928990,"num_write_kb":7715954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15808937984,"data_stored":24991301670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45612,"ondisk_log_size":45612,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739236,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9728296,"kb_used_data":7722592,"kb_used_omap":1043,"kb_used_meta":2004652,"kb_avail":84643544,"statfs":{"total":96636764160,"available":86674989056,"internally_reserved":0,"allocated":7907934208,"data_stored":12498309700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":2052763895},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.34899999999999998,"15min":0.45100000000000001},"min":{"1min":0.19600000000000001,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":4.2560000000000002,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.52500000000000002,"15min":0.56200000000000006},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.183},"max":{"1min":4.1509999999999998,"5min":5.4050000000000002,"15min":27.538},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:16:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48699999999999999,"15min":0.437},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.16700000000000001},"max":{"1min":1.994,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.47899999999999998,"15min":0.44},"min":{"1min":0.22700000000000001,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.298,"5min":4.298,"15min":4.298},"last":0.40200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739234,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5562556,"kb_used_data":3760372,"kb_used_omap":472,"kb_used_meta":1801703,"kb_avail":88809284,"statfs":{"total":96636764160,"available":90940706816,"internally_reserved":0,"allocated":3850620928,"data_stored":6091071176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1844944591},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,5,3,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.57099999999999995,"15min":0.46400000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.16900000000000001},"max":{"1min":2.3780000000000001,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.222,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.83299999999999996,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":1.1459999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:16:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.47699999999999998,"15min":0.499},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":4.5170000000000003,"5min":4.5170000000000003,"15min":15.891},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.54500000000000004,"15min":0.51100000000000001},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":0.621,"5min":5.835,"15min":5.835},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5584552,"kb_used_data":3958752,"kb_used_omap":500,"kb_used_meta":1625291,"kb_avail":88787288,"statfs":{"total":96636764160,"available":90918182912,"internally_reserved":0,"allocated":4053762048,"data_stored":6402614885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1664298597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,4,7,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.47399999999999998},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.156},"max":{"1min":1.212,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.45000000000000001},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.16800000000000001},"max":{"1min":3.3109999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54000000000000004,"15min":0.46999999999999997},"min":{"1min":0.21199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.1539999999999999,"5min":3.1629999999999998,"15min":5.1909999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.53500000000000003,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":0.79200000000000004,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4052267008,"data_stored":6401994752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3848892416,"data_stored":6090346515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7907778560,"data_stored":12498960403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:17:14.689 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:14.689 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:14.690 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:14.690 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:14.691 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:29.490 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:17:29.489+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:17:34.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:35.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:17:35.124 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:17:35.165 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2293,"stamp":"2023-10-14T01:17:33.481480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27336117989,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1972595,"num_read_kb":7662074,"num_write":1937407,"num_write_kb":7748687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85926,"ondisk_log_size":85926,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21113416,"kb_used_data":15502832,"kb_used_omap":2016,"kb_used_meta":5608543,"kb_avail":262002104,"statfs":{"total":289910292480,"available":268290154496,"internally_reserved":0,"allocated":15874899968,"data_stored":25054578545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5743148587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,6,11,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14774272,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3916,"num_read_kb":15596,"num_write":3793,"num_write_kb":15172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002050"},"pg_stats":[{"pgid":"2.7","version":"123'6450","reported_seq":9550,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:26.895539+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:17:26.895539+0000","last_peered":"2023-10-14T01:17:26.895539+0000","last_clean":"2023-10-14T01:17:26.895539+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:17:26.895539+0000","last_undegraded":"2023-10-14T01:17:26.895539+0000","last_fullsized":"2023-10-14T01:17:26.895539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"123'9598","reported_seq":15425,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:26.895272+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:17:26.895272+0000","last_peered":"2023-10-14T01:17:26.895272+0000","last_clean":"2023-10-14T01:17:26.895272+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:17:26.895272+0000","last_undegraded":"2023-10-14T01:17:26.895272+0000","last_fullsized":"2023-10-14T01:17:26.895272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9598,"log_dups_size":0,"ondisk_log_size":9598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"123'6423","reported_seq":9322,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:26.894984+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:17:26.894984+0000","last_peered":"2023-10-14T01:17:26.894984+0000","last_clean":"2023-10-14T01:17:26.894984+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:17:26.894984+0000","last_undegraded":"2023-10-14T01:17:26.894984+0000","last_fullsized":"2023-10-14T01:17:26.894984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6423,"log_dups_size":0,"ondisk_log_size":6423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"123'15830","reported_seq":23544,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:26.896060+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:17:26.896060+0000","last_peered":"2023-10-14T01:17:26.896060+0000","last_clean":"2023-10-14T01:17:26.896060+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:17:26.896060+0000","last_undegraded":"2023-10-14T01:17:26.896060+0000","last_fullsized":"2023-10-14T01:17:26.896060+0000","mapping_epoch":12,"log_start":"85'5800","ondisk_log_start":"85'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"123'503545","reported_seq":1001426,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:29.470355+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:17:29.470355+0000","last_peered":"2023-10-14T01:17:29.470355+0000","last_clean":"2023-10-14T01:17:29.470355+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:17:29.470355+0000","last_undegraded":"2023-10-14T01:17:29.470355+0000","last_fullsized":"2023-10-14T01:17:29.470355+0000","mapping_epoch":15,"log_start":"122'493535","ondisk_log_start":"122'493535","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1742,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 344s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7060815872,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":500024,"num_read_kb":1969012,"num_write":499827,"num_write_kb":1999308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"123'3336","reported_seq":4969,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:26.895202+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:17:26.895202+0000","last_peered":"2023-10-14T01:17:26.895202+0000","last_clean":"2023-10-14T01:17:26.895202+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:17:26.895202+0000","last_undegraded":"2023-10-14T01:17:26.895202+0000","last_fullsized":"2023-10-14T01:17:26.895202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3336,"log_dups_size":0,"ondisk_log_size":3336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"123'477065","reported_seq":950338,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:28.878375+0000","last_change":"2023-10-14T01:17:25.637415+0000","last_active":"2023-10-14T01:17:28.878375+0000","last_peered":"2023-10-14T01:17:28.878375+0000","last_clean":"2023-10-14T01:17:28.878375+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:17:28.878375+0000","last_undegraded":"2023-10-14T01:17:28.878375+0000","last_fullsized":"2023-10-14T01:17:28.878375+0000","mapping_epoch":15,"log_start":"122'467023","ondisk_log_start":"122'467023","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6705967104,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":475366,"num_read_kb":1871420,"num_write":474669,"num_write_kb":1898676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2945,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:32.030285+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:17:32.030285+0000","last_peered":"2023-10-14T01:17:32.030285+0000","last_clean":"2023-10-14T01:17:32.030285+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:17:32.030285+0000","last_undegraded":"2023-10-14T01:17:32.030285+0000","last_fullsized":"2023-10-14T01:17:32.030285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"123'497123","reported_seq":988908,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:29.453859+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:17:29.453859+0000","last_peered":"2023-10-14T01:17:29.453859+0000","last_clean":"2023-10-14T01:17:29.453859+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:17:29.453859+0000","last_undegraded":"2023-10-14T01:17:29.453859+0000","last_fullsized":"2023-10-14T01:17:29.453859+0000","mapping_epoch":15,"log_start":"122'487028","ondisk_log_start":"122'487028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1912,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 345s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6955548672,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":493855,"num_read_kb":1944556,"num_write":493340,"num_write_kb":1973360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3095,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:32.022772+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:17:32.022772+0000","last_peered":"2023-10-14T01:17:32.022772+0000","last_clean":"2023-10-14T01:17:32.022772+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:17:32.022772+0000","last_undegraded":"2023-10-14T01:17:32.022772+0000","last_fullsized":"2023-10-14T01:17:32.022772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"123'472497","reported_seq":940149,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:28.860196+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:17:28.860196+0000","last_peered":"2023-10-14T01:17:28.860196+0000","last_clean":"2023-10-14T01:17:28.860196+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:17:28.860196+0000","last_undegraded":"2023-10-14T01:17:28.860196+0000","last_fullsized":"2023-10-14T01:17:28.860196+0000","mapping_epoch":15,"log_start":"122'462425","ondisk_log_start":"122'462425","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2007,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 323s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6613192723,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":469602,"num_read_kb":1848280,"num_write":468775,"num_write_kb":1875094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"123'9547","reported_seq":13790,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:26.895647+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:17:26.895647+0000","last_peered":"2023-10-14T01:17:26.895647+0000","last_clean":"2023-10-14T01:17:26.895647+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:17:26.895647+0000","last_undegraded":"2023-10-14T01:17:26.895647+0000","last_fullsized":"2023-10-14T01:17:26.895647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9547,"log_dups_size":0,"ondisk_log_size":9547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":296,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-14T01:15:38.355507+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:15:38.355507+0000","last_peered":"2023-10-14T01:15:38.355507+0000","last_clean":"2023-10-14T01:15:38.355507+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:15:38.355507+0000","last_undegraded":"2023-10-14T01:15:38.355507+0000","last_fullsized":"2023-10-14T01:15:38.355507+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27335524371,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1938847,"num_read_kb":7633268,"num_write":1936611,"num_write_kb":7746438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15871295488,"data_stored":25053659174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40219,"ondisk_log_size":40219,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45608,"ondisk_log_size":45608,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9854800,"kb_used_data":7753160,"kb_used_omap":1043,"kb_used_meta":2100588,"kb_avail":84517040,"statfs":{"total":96636764160,"available":86545448960,"internally_reserved":0,"allocated":7939235840,"data_stored":12529611332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":2151002359},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.34899999999999998,"15min":0.45100000000000001},"min":{"1min":0.19600000000000001,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":4.2560000000000002,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.182},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.52500000000000002,"15min":0.56200000000000006},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.183},"max":{"1min":4.1509999999999998,"5min":5.4050000000000002,"15min":27.538},"last":0.35099999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:16:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48699999999999999,"15min":0.437},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.16700000000000001},"max":{"1min":1.994,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.47899999999999998,"15min":0.44},"min":{"1min":0.22700000000000001,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.298,"5min":4.298,"15min":4.298},"last":0.33700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739238,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5634068,"kb_used_data":3775628,"kb_used_omap":472,"kb_used_meta":1857959,"kb_avail":88737772,"statfs":{"total":96636764160,"available":90867478528,"internally_reserved":0,"allocated":3866243072,"data_stored":6106693320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1902550735},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.57099999999999995,"15min":0.46400000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.16900000000000001},"max":{"1min":2.3780000000000001,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.222,"5min":0.22,"15min":0.20200000000000001},"max":{"1min":0.83299999999999996,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:16:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.47699999999999998,"15min":0.499},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":4.5170000000000003,"5min":4.5170000000000003,"15min":15.891},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.54500000000000004,"15min":0.51100000000000001},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.21299999999999999},"max":{"1min":0.621,"5min":5.835,"15min":5.835},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359739237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5624548,"kb_used_data":3974044,"kb_used_omap":500,"kb_used_meta":1649995,"kb_avail":88747292,"statfs":{"total":96636764160,"available":90877227008,"internally_reserved":0,"allocated":4069421056,"data_stored":6418273893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1689595493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.47399999999999998},"min":{"1min":0.188,"5min":0.17699999999999999,"15min":0.156},"max":{"1min":1.212,"5min":5.7729999999999997,"15min":5.7729999999999997},"last":0.216},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.45000000000000001},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.16800000000000001},"max":{"1min":3.3109999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":1.248}]},{"osd":2,"last update":"Sat Oct 14 01:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54000000000000004,"15min":0.46999999999999997},"min":{"1min":0.21199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.1539999999999999,"5min":3.1629999999999998,"15min":5.1909999999999998},"last":1.998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.53500000000000003,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.21199999999999999,"15min":0.20699999999999999},"max":{"1min":0.79200000000000004,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":2.048}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4068024320,"data_stored":6417752064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3864354816,"data_stored":6105808915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7938916352,"data_stored":12530098195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:17:35.167 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:35.167 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:35.168 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:35.168 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:35.169 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:37.975 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:37.973+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:17:37.995 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:37.993+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:17:47.013 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:46.997+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:17:47.014 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:46.997+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:17:47.014 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:46.997+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:17:47.014 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:47.005+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:17:47.015 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:17:47.005+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:17:55.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:55.876 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:17:55.877 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:17:55.932 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2305,"stamp":"2023-10-14T01:17:55.487558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27372007141,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1981768,"num_read_kb":7698590,"num_write":1946659,"num_write_kb":7785695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86079,"ondisk_log_size":86079,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21372040,"kb_used_data":15568944,"kb_used_omap":2016,"kb_used_meta":5801055,"kb_avail":261743480,"statfs":{"total":289910292480,"available":268025323520,"internally_reserved":0,"allocated":15942598656,"data_stored":25122253935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5940280875},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,0,11,5,10,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21553152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5533,"num_read_kb":22020,"num_write":5572,"num_write_kb":22288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004410"},"pg_stats":[{"pgid":"2.7","version":"124'6496","reported_seq":9607,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.496050+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:17:52.496050+0000","last_peered":"2023-10-14T01:17:52.496050+0000","last_clean":"2023-10-14T01:17:52.496050+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:17:52.496050+0000","last_undegraded":"2023-10-14T01:17:52.496050+0000","last_fullsized":"2023-10-14T01:17:52.496050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6496,"log_dups_size":0,"ondisk_log_size":6496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'9666","reported_seq":15509,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.495506+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:17:52.495506+0000","last_peered":"2023-10-14T01:17:52.495506+0000","last_clean":"2023-10-14T01:17:52.495506+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:17:52.495506+0000","last_undegraded":"2023-10-14T01:17:52.495506+0000","last_fullsized":"2023-10-14T01:17:52.495506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"124'6470","reported_seq":9380,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.495464+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:17:52.495464+0000","last_peered":"2023-10-14T01:17:52.495464+0000","last_clean":"2023-10-14T01:17:52.495464+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:17:52.495464+0000","last_undegraded":"2023-10-14T01:17:52.495464+0000","last_fullsized":"2023-10-14T01:17:52.495464+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'15942","reported_seq":23682,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.496287+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:17:52.496287+0000","last_peered":"2023-10-14T01:17:52.496287+0000","last_clean":"2023-10-14T01:17:52.496287+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:17:52.496287+0000","last_undegraded":"2023-10-14T01:17:52.496287+0000","last_fullsized":"2023-10-14T01:17:52.496287+0000","mapping_epoch":12,"log_start":"85'5900","ondisk_log_start":"85'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'506051","reported_seq":1006383,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:54.628735+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:17:54.628735+0000","last_peered":"2023-10-14T01:17:54.628735+0000","last_clean":"2023-10-14T01:17:54.628735+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:17:54.628735+0000","last_undegraded":"2023-10-14T01:17:54.628735+0000","last_fullsized":"2023-10-14T01:17:54.628735+0000","mapping_epoch":15,"log_start":"122'496035","ondisk_log_start":"122'496035","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1910,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 369s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7070146560,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":502457,"num_read_kb":1978716,"num_write":502218,"num_write_kb":2008872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"124'3353","reported_seq":4992,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.855180+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:17:52.855180+0000","last_peered":"2023-10-14T01:17:52.855180+0000","last_clean":"2023-10-14T01:17:52.855180+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:17:52.855180+0000","last_undegraded":"2023-10-14T01:17:52.855180+0000","last_fullsized":"2023-10-14T01:17:52.855180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'479333","reported_seq":954902,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:53.989791+0000","last_change":"2023-10-14T01:17:53.685514+0000","last_active":"2023-10-14T01:17:53.989791+0000","last_peered":"2023-10-14T01:17:53.989791+0000","last_clean":"2023-10-14T01:17:53.989791+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:17:53.989791+0000","last_undegraded":"2023-10-14T01:17:53.989791+0000","last_fullsized":"2023-10-14T01:17:53.989791+0000","mapping_epoch":15,"log_start":"122'469323","ondisk_log_start":"122'469323","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6714781696,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":477653,"num_read_kb":1880500,"num_write":476937,"num_write_kb":1907748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2954,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.546762+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:17:52.546762+0000","last_peered":"2023-10-14T01:17:52.546762+0000","last_clean":"2023-10-14T01:17:52.546762+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:17:52.546762+0000","last_undegraded":"2023-10-14T01:17:52.546762+0000","last_fullsized":"2023-10-14T01:17:52.546762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'499600","reported_seq":993677,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:54.625750+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:17:54.625750+0000","last_peered":"2023-10-14T01:17:54.625750+0000","last_clean":"2023-10-14T01:17:54.625750+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:17:54.625750+0000","last_undegraded":"2023-10-14T01:17:54.625750+0000","last_fullsized":"2023-10-14T01:17:54.625750+0000","mapping_epoch":15,"log_start":"122'489528","ondisk_log_start":"122'489528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2077,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 370s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6964649984,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":496129,"num_read_kb":1953616,"num_write":495703,"num_write_kb":1982812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3105,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.497269+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:17:52.497269+0000","last_peered":"2023-10-14T01:17:52.497269+0000","last_clean":"2023-10-14T01:17:52.497269+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:17:52.497269+0000","last_undegraded":"2023-10-14T01:17:52.497269+0000","last_fullsized":"2023-10-14T01:17:52.497269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'474841","reported_seq":944687,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:53.998848+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:17:53.998848+0000","last_peered":"2023-10-14T01:17:53.998848+0000","last_clean":"2023-10-14T01:17:53.998848+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:17:53.998848+0000","last_undegraded":"2023-10-14T01:17:53.998848+0000","last_fullsized":"2023-10-14T01:17:53.998848+0000","mapping_epoch":15,"log_start":"122'464825","ondisk_log_start":"122'464825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2175,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6621835283,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":471781,"num_read_kb":1856952,"num_write":471005,"num_write_kb":1884014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"124'9615","reported_seq":13874,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:52.496275+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:17:52.496275+0000","last_peered":"2023-10-14T01:17:52.496275+0000","last_clean":"2023-10-14T01:17:52.496275+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:17:52.496275+0000","last_undegraded":"2023-10-14T01:17:52.496275+0000","last_fullsized":"2023-10-14T01:17:52.496275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9615,"log_dups_size":0,"ondisk_log_size":9615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":298,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:38.949301+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:17:38.949301+0000","last_peered":"2023-10-14T01:17:38.949301+0000","last_clean":"2023-10-14T01:17:38.949301+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:17:38.949301+0000","last_undegraded":"2023-10-14T01:17:38.949301+0000","last_fullsized":"2023-10-14T01:17:38.949301+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27371413523,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1948020,"num_read_kb":7669784,"num_write":1945863,"num_write_kb":7783446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15939239936,"data_stored":25121603622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40114,"ondisk_log_size":40114,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":45866,"ondisk_log_size":45866,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739244,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9987980,"kb_used_data":7782276,"kb_used_omap":1043,"kb_used_meta":2204652,"kb_avail":84383860,"statfs":{"total":96636764160,"available":86409072640,"internally_reserved":0,"allocated":7969050624,"data_stored":12559418350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":2257563895},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:36 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.34899999999999998,"15min":0.439},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.45300000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.53600000000000003,"15min":0.54700000000000004},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":1.1899999999999999,"5min":5.4050000000000002,"15min":27.538},"last":0.442}]},{"osd":1,"last update":"Sat Oct 14 01:16:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48699999999999999,"15min":0.437},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.16700000000000001},"max":{"1min":1.994,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.47899999999999998,"15min":0.44},"min":{"1min":0.22700000000000001,"5min":0.20300000000000001,"15min":0.186},"max":{"1min":4.298,"5min":4.298,"15min":4.298},"last":0.496}]}]},{"osd":1,"up_from":8,"seq":34359739243,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5675468,"kb_used_data":3793476,"kb_used_omap":472,"kb_used_meta":1881511,"kb_avail":88696372,"statfs":{"total":96636764160,"available":90825084928,"internally_reserved":0,"allocated":3884519424,"data_stored":6124961906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1926667983},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.53700000000000003,"15min":0.46500000000000002},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":0.505,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.315},{"interface":"front","average":{"1min":0.44,"5min":0.497,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":1.1459999999999999,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.48399999999999999},"min":{"1min":0.20499999999999999,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":1.133,"5min":4.5170000000000003,"15min":15.891},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.51100000000000001,"15min":0.49199999999999999},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.55400000000000005,"5min":5.835,"15min":5.835},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5708592,"kb_used_data":3993192,"kb_used_omap":500,"kb_used_meta":1714891,"kb_avail":88663248,"statfs":{"total":96636764160,"available":90791165952,"internally_reserved":0,"allocated":4089028608,"data_stored":6437873679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1756048997},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,2,1,6,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39200000000000002,"15min":0.45300000000000001},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.16600000000000001},"max":{"1min":1.8220000000000001,"5min":4.1539999999999999,"15min":5.7729999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.46400000000000002,"15min":0.45100000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":1.2749999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.46600000000000003},"min":{"1min":0.20699999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.998,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53400000000000003,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":2.048,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.46899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4087492608,"data_stored":6437220352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3882766336,"data_stored":6124220435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7968980992,"data_stored":12560162835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:17:55.934 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:55.934 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:55.934 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:55.935 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:55.935 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:15.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:16.482 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:18:16.483 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:18:16.509 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2315,"stamp":"2023-10-14T01:18:15.491046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27400797925,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1989088,"num_read_kb":7727718,"num_write":1954095,"num_write_kb":7815439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86314,"ondisk_log_size":86314,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21388776,"kb_used_data":15629648,"kb_used_omap":2016,"kb_used_meta":5757087,"kb_avail":261726744,"statfs":{"total":289910292480,"available":268008185856,"internally_reserved":0,"allocated":16004759552,"data_stored":25184414831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":5895257643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,7,7,16,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21917696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5590,"num_read_kb":22260,"num_write":5659,"num_write_kb":22636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002062"},"pg_stats":[{"pgid":"2.7","version":"124'6518","reported_seq":9637,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.975274+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:18:12.975274+0000","last_peered":"2023-10-14T01:18:12.975274+0000","last_clean":"2023-10-14T01:18:12.975274+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:18:12.975274+0000","last_undegraded":"2023-10-14T01:18:12.975274+0000","last_fullsized":"2023-10-14T01:18:12.975274+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6518,"log_dups_size":0,"ondisk_log_size":6518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'9699","reported_seq":15554,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.975194+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:18:12.975194+0000","last_peered":"2023-10-14T01:18:12.975194+0000","last_clean":"2023-10-14T01:18:12.975194+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:18:12.975194+0000","last_undegraded":"2023-10-14T01:18:12.975194+0000","last_fullsized":"2023-10-14T01:18:12.975194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"124'6492","reported_seq":9410,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.974938+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:18:12.974938+0000","last_peered":"2023-10-14T01:18:12.974938+0000","last_clean":"2023-10-14T01:18:12.974938+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:18:12.974938+0000","last_undegraded":"2023-10-14T01:18:12.974938+0000","last_fullsized":"2023-10-14T01:18:12.974938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6492,"log_dups_size":0,"ondisk_log_size":6492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'15997","reported_seq":23757,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.975676+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:18:12.975676+0000","last_peered":"2023-10-14T01:18:12.975676+0000","last_clean":"2023-10-14T01:18:12.975676+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:18:12.975676+0000","last_undegraded":"2023-10-14T01:18:12.975676+0000","last_fullsized":"2023-10-14T01:18:12.975676+0000","mapping_epoch":12,"log_start":"85'5900","ondisk_log_start":"85'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'508051","reported_seq":1010352,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:14.733951+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:18:14.733951+0000","last_peered":"2023-10-14T01:18:14.733951+0000","last_clean":"2023-10-14T01:18:14.733951+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:18:14.733951+0000","last_undegraded":"2023-10-14T01:18:14.733951+0000","last_fullsized":"2023-10-14T01:18:14.733951+0000","mapping_epoch":15,"log_start":"123'498036","ondisk_log_start":"123'498036","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2005,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 389s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7077687296,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":504418,"num_read_kb":1986516,"num_write":504152,"num_write_kb":2016608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"124'3365","reported_seq":5008,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.831423+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:18:12.831423+0000","last_peered":"2023-10-14T01:18:12.831423+0000","last_clean":"2023-10-14T01:18:12.831423+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:18:12.831423+0000","last_undegraded":"2023-10-14T01:18:12.831423+0000","last_fullsized":"2023-10-14T01:18:12.831423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3365,"log_dups_size":0,"ondisk_log_size":3365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'481168","reported_seq":958520,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:14.096476+0000","last_change":"2023-10-14T01:18:09.838661+0000","last_active":"2023-10-14T01:18:14.096476+0000","last_peered":"2023-10-14T01:18:14.096476+0000","last_clean":"2023-10-14T01:18:14.096476+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:18:14.096476+0000","last_undegraded":"2023-10-14T01:18:14.096476+0000","last_fullsized":"2023-10-14T01:18:14.096476+0000","mapping_epoch":15,"log_start":"122'471123","ondisk_log_start":"122'471123","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6721921024,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":479427,"num_read_kb":1887564,"num_write":478772,"num_write_kb":1915088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2963,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.976344+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:18:12.976344+0000","last_peered":"2023-10-14T01:18:12.976344+0000","last_clean":"2023-10-14T01:18:12.976344+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:18:12.976344+0000","last_undegraded":"2023-10-14T01:18:12.976344+0000","last_fullsized":"2023-10-14T01:18:12.976344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'501525","reported_seq":997479,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:14.738218+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:18:14.738218+0000","last_peered":"2023-10-14T01:18:14.738218+0000","last_clean":"2023-10-14T01:18:14.738218+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:18:14.738218+0000","last_undegraded":"2023-10-14T01:18:14.738218+0000","last_fullsized":"2023-10-14T01:18:14.738218+0000","mapping_epoch":15,"log_start":"122'491428","ondisk_log_start":"122'491428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2181,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 390s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6971813888,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":497992,"num_read_kb":1961028,"num_write":497556,"num_write_kb":1990224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3113,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.977606+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:18:12.977606+0000","last_peered":"2023-10-14T01:18:12.977606+0000","last_clean":"2023-10-14T01:18:12.977606+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:18:12.977606+0000","last_undegraded":"2023-10-14T01:18:12.977606+0000","last_fullsized":"2023-10-14T01:18:12.977606+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'476740","reported_seq":948323,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:14.100079+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:18:14.100079+0000","last_peered":"2023-10-14T01:18:14.100079+0000","last_clean":"2023-10-14T01:18:14.100079+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:18:14.100079+0000","last_undegraded":"2023-10-14T01:18:14.100079+0000","last_fullsized":"2023-10-14T01:18:14.100079+0000","mapping_epoch":15,"log_start":"122'466725","ondisk_log_start":"122'466725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2297,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 368s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6628782099,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":473503,"num_read_kb":1863804,"num_write":472819,"num_write_kb":1891270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"124'9648","reported_seq":13919,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:12.975525+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:18:12.975525+0000","last_peered":"2023-10-14T01:18:12.975525+0000","last_clean":"2023-10-14T01:18:12.975525+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:18:12.975525+0000","last_undegraded":"2023-10-14T01:18:12.975525+0000","last_fullsized":"2023-10-14T01:18:12.975525+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9648,"log_dups_size":0,"ondisk_log_size":9648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":298,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:38.949301+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:17:38.949301+0000","last_peered":"2023-10-14T01:17:38.949301+0000","last_clean":"2023-10-14T01:17:38.949301+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:17:38.949301+0000","last_undegraded":"2023-10-14T01:17:38.949301+0000","last_fullsized":"2023-10-14T01:17:38.949301+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27400204307,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1955340,"num_read_kb":7698912,"num_write":1953299,"num_write_kb":7813190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16000274432,"data_stored":25182638118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46043,"ondisk_log_size":46043,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739248,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9866612,"kb_used_data":7813164,"kb_used_omap":1043,"kb_used_meta":2052396,"kb_avail":84505228,"statfs":{"total":96636764160,"available":86533353472,"internally_reserved":0,"allocated":8000679936,"data_stored":12591047662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":2101653751},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:36 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.34899999999999998,"15min":0.439},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.45300000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.53600000000000003,"15min":0.54700000000000004},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":1.1899999999999999,"5min":5.4050000000000002,"15min":27.538},"last":0.26300000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52100000000000002,"15min":0.44800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23000000000000001,"15min":0.16700000000000001},"max":{"1min":3.6949999999999998,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.50700000000000001,"15min":0.438},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.186},"max":{"1min":3.214,"5min":4.298,"15min":4.298},"last":0.28399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739247,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5762380,"kb_used_data":3808196,"kb_used_omap":472,"kb_used_meta":1953703,"kb_avail":88609460,"statfs":{"total":96636764160,"available":90736087040,"internally_reserved":0,"allocated":3899592704,"data_stored":6140035186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":2000592591},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.53700000000000003,"15min":0.46500000000000002},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":0.505,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.497,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":1.1459999999999999,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.48399999999999999},"min":{"1min":0.20499999999999999,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":1.133,"5min":4.5170000000000003,"15min":15.891},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.51100000000000001,"15min":0.49199999999999999},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.55400000000000005,"5min":5.835,"15min":5.835},"last":0.40300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739246,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5759784,"kb_used_data":4008288,"kb_used_omap":500,"kb_used_meta":1750987,"kb_avail":88612056,"statfs":{"total":96636764160,"available":90738745344,"internally_reserved":0,"allocated":4104486912,"data_stored":6453331983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1793011301},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,6,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39200000000000002,"15min":0.45300000000000001},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.16600000000000001},"max":{"1min":1.8220000000000001,"5min":4.1539999999999999,"15min":5.7729999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.46400000000000002,"15min":0.45100000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":1.2749999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":2.6120000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.46600000000000003},"min":{"1min":0.20699999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.998,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.249},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53400000000000003,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":2.048,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4102995968,"data_stored":6452723712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3897712640,"data_stored":6139166739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7999565824,"data_stored":12590747667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:18:16.510 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:16.511 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:16.511 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:16.512 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:16.512 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:34.279 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:18:34.281 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:18:34.282 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:18:34.282 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:18:34.282 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:18:34.283 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:18:34.283 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:18:34.283 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:18:34.283 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:18:34.284 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:18:34.284 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:18:34.284 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:18:34.285 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:18:34.285 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:18:34.285 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:18:34.272+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:18:36.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:36.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:18:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:18:36.994 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2325,"stamp":"2023-10-14T01:18:35.494381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27430608613,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":1996454,"num_read_kb":7757062,"num_write":1961798,"num_write_kb":7846251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86433,"ondisk_log_size":86433,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21632920,"kb_used_data":15690944,"kb_used_omap":2016,"kb_used_meta":5939935,"kb_avail":261482600,"statfs":{"total":289910292480,"available":267758182400,"internally_reserved":0,"allocated":16067526656,"data_stored":25247181935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2064853,"internal_metadata":6082493995},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,7,16,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22323200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5578,"num_read_kb":22224,"num_write":5771,"num_write_kb":23084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001983"},"pg_stats":[{"pgid":"2.7","version":"124'6542","reported_seq":9669,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.263385+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:18:34.263385+0000","last_peered":"2023-10-14T01:18:34.263385+0000","last_clean":"2023-10-14T01:18:34.263385+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:18:34.263385+0000","last_undegraded":"2023-10-14T01:18:34.263385+0000","last_fullsized":"2023-10-14T01:18:34.263385+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'9735","reported_seq":15601,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.258875+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:18:34.258875+0000","last_peered":"2023-10-14T01:18:34.258875+0000","last_clean":"2023-10-14T01:18:34.258875+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:18:34.258875+0000","last_undegraded":"2023-10-14T01:18:34.258875+0000","last_fullsized":"2023-10-14T01:18:34.258875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9735,"log_dups_size":0,"ondisk_log_size":9735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"124'6516","reported_seq":9442,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.261067+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:18:34.261067+0000","last_peered":"2023-10-14T01:18:34.261067+0000","last_clean":"2023-10-14T01:18:34.261067+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:18:34.261067+0000","last_undegraded":"2023-10-14T01:18:34.261067+0000","last_fullsized":"2023-10-14T01:18:34.261067+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'16052","reported_seq":23832,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:33.460137+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:18:33.460137+0000","last_peered":"2023-10-14T01:18:33.460137+0000","last_clean":"2023-10-14T01:18:33.460137+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:18:33.460137+0000","last_undegraded":"2023-10-14T01:18:33.460137+0000","last_fullsized":"2023-10-14T01:18:33.460137+0000","mapping_epoch":12,"log_start":"87'6000","ondisk_log_start":"87'6000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'510140","reported_seq":1014377,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:34.901595+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:18:34.901595+0000","last_peered":"2023-10-14T01:18:34.901595+0000","last_clean":"2023-10-14T01:18:34.901595+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:18:34.901595+0000","last_undegraded":"2023-10-14T01:18:34.901595+0000","last_fullsized":"2023-10-14T01:18:34.901595+0000","mapping_epoch":15,"log_start":"123'500136","ondisk_log_start":"123'500136","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2083,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 409s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7085678592,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":506340,"num_read_kb":1994164,"num_write":506187,"num_write_kb":2024748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"124'3378","reported_seq":5025,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:33.459018+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:18:33.459018+0000","last_peered":"2023-10-14T01:18:33.459018+0000","last_clean":"2023-10-14T01:18:33.459018+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:18:33.459018+0000","last_undegraded":"2023-10-14T01:18:33.459018+0000","last_fullsized":"2023-10-14T01:18:33.459018+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'483076","reported_seq":962255,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.213036+0000","last_change":"2023-10-14T01:18:31.041143+0000","last_active":"2023-10-14T01:18:34.213036+0000","last_peered":"2023-10-14T01:18:34.213036+0000","last_clean":"2023-10-14T01:18:34.213036+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:18:34.213036+0000","last_undegraded":"2023-10-14T01:18:34.213036+0000","last_fullsized":"2023-10-14T01:18:34.213036+0000","mapping_epoch":15,"log_start":"123'473023","ondisk_log_start":"123'473023","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6729297920,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":481245,"num_read_kb":1894792,"num_write":480680,"num_write_kb":1922720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2971,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:33.474343+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:18:33.474343+0000","last_peered":"2023-10-14T01:18:33.474343+0000","last_clean":"2023-10-14T01:18:33.474343+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:18:33.474343+0000","last_undegraded":"2023-10-14T01:18:33.474343+0000","last_fullsized":"2023-10-14T01:18:33.474343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'503524","reported_seq":1001376,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:34.900937+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:18:34.900937+0000","last_peered":"2023-10-14T01:18:34.900937+0000","last_clean":"2023-10-14T01:18:34.900937+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:18:34.900937+0000","last_undegraded":"2023-10-14T01:18:34.900937+0000","last_fullsized":"2023-10-14T01:18:34.900937+0000","mapping_epoch":15,"log_start":"123'493428","ondisk_log_start":"123'493428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2292,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 410s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6979129344,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":499879,"num_read_kb":1968556,"num_write":499478,"num_write_kb":1997912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3121,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:33.464210+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:18:33.464210+0000","last_peered":"2023-10-14T01:18:33.464210+0000","last_clean":"2023-10-14T01:18:33.464210+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:18:33.464210+0000","last_undegraded":"2023-10-14T01:18:33.464210+0000","last_fullsized":"2023-10-14T01:18:33.464210+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'478678","reported_seq":952013,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:34.241341+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:18:34.241341+0000","last_peered":"2023-10-14T01:18:34.241341+0000","last_clean":"2023-10-14T01:18:34.241341+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:18:34.241341+0000","last_undegraded":"2023-10-14T01:18:34.241341+0000","last_fullsized":"2023-10-14T01:18:34.241341+0000","mapping_epoch":15,"log_start":"123'468625","ondisk_log_start":"123'468625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2438,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 388s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6635909139,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":475242,"num_read_kb":1870744,"num_write":474657,"num_write_kb":1898622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"124'9681","reported_seq":13964,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:33.459986+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:18:33.459986+0000","last_peered":"2023-10-14T01:18:33.459986+0000","last_clean":"2023-10-14T01:18:33.459986+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:18:33.459986+0000","last_undegraded":"2023-10-14T01:18:33.459986+0000","last_fullsized":"2023-10-14T01:18:33.459986+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":298,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:38.949301+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:17:38.949301+0000","last_peered":"2023-10-14T01:17:38.949301+0000","last_clean":"2023-10-14T01:17:38.949301+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:17:38.949301+0000","last_undegraded":"2023-10-14T01:17:38.949301+0000","last_fullsized":"2023-10-14T01:17:38.949301+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27430014995,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1962706,"num_read_kb":7728256,"num_write":1961002,"num_write_kb":7844002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16063373312,"data_stored":25245736998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40206,"ondisk_log_size":40206,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46128,"ondisk_log_size":46128,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739252,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9983236,"kb_used_data":7843772,"kb_used_omap":1043,"kb_used_meta":2138412,"kb_avail":84388604,"statfs":{"total":96636764160,"available":86413930496,"internally_reserved":0,"allocated":8032022528,"data_stored":12622390254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1068809,"internal_metadata":2189734135},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:36 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.34899999999999998,"15min":0.439},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.159},"max":{"1min":0.45300000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.53600000000000003,"15min":0.54700000000000004},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.183},"max":{"1min":1.1899999999999999,"5min":5.4050000000000002,"15min":27.538},"last":0.32300000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52100000000000002,"15min":0.44800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23000000000000001,"15min":0.16700000000000001},"max":{"1min":3.6949999999999998,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.50700000000000001,"15min":0.438},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.186},"max":{"1min":3.214,"5min":4.298,"15min":4.298},"last":0.30299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739251,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5813408,"kb_used_data":3823192,"kb_used_omap":472,"kb_used_meta":1989735,"kb_avail":88558432,"statfs":{"total":96636764160,"available":90683834368,"internally_reserved":0,"allocated":3914948608,"data_stored":6155391090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":2037489359},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.53700000000000003,"15min":0.46500000000000002},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":0.505,"5min":5.7869999999999999,"15min":5.7869999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.497,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":1.1459999999999999,"5min":5.8979999999999997,"15min":6.9199999999999999},"last":0.25}]},{"osd":2,"last update":"Sat Oct 14 01:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.48399999999999999},"min":{"1min":0.20499999999999999,"5min":0.20000000000000001,"15min":0.182},"max":{"1min":1.133,"5min":4.5170000000000003,"15min":15.891},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.51100000000000001,"15min":0.49199999999999999},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.55400000000000005,"5min":5.835,"15min":5.835},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5836276,"kb_used_data":4023980,"kb_used_omap":500,"kb_used_meta":1811787,"kb_avail":88535564,"statfs":{"total":96636764160,"available":90660417536,"internally_reserved":0,"allocated":4120555520,"data_stored":6469400591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1855270501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,8,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39200000000000002,"15min":0.45300000000000001},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.16600000000000001},"max":{"1min":1.8220000000000001,"5min":4.1539999999999999,"15min":5.7729999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.46400000000000002,"15min":0.45100000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":1.2749999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.314}]},{"osd":2,"last update":"Sat Oct 14 01:17:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53700000000000003,"15min":0.46600000000000003},"min":{"1min":0.20699999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.998,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.53400000000000003,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":2.048,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4119203840,"data_stored":6468931584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3913072640,"data_stored":6154526739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8031096832,"data_stored":12622278675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:18:36.997 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:36.998 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:36.998 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:36.999 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:37.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:18:37.528 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:18:37.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:18:38.035 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:18:38.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:18:38.583 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:18:38.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:18:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:18:39.209 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:41.288 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:18:41.284+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:18:59.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:59.790 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:18:59.791 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:18:59.825 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2337,"stamp":"2023-10-14T01:18:59.498428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27462803173,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2004372,"num_read_kb":7788614,"num_write":1970105,"num_write_kb":7879479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86452,"ondisk_log_size":86452,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20206148,"kb_used_data":15760876,"kb_used_omap":2243,"kb_used_meta":4443004,"kb_avail":262909372,"statfs":{"total":289910292480,"available":269219196928,"internally_reserved":0,"allocated":16139137024,"data_stored":25318792303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2296904,"internal_metadata":4549637048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,6,8,16,14,19,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18116608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4437,"num_read_kb":17684,"num_write":4680,"num_write_kb":18720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001956"},"pg_stats":[{"pgid":"2.7","version":"124'6566","reported_seq":9703,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:59.055309+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:18:59.055309+0000","last_peered":"2023-10-14T01:18:59.055309+0000","last_clean":"2023-10-14T01:18:59.055309+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:18:59.055309+0000","last_undegraded":"2023-10-14T01:18:59.055309+0000","last_fullsized":"2023-10-14T01:18:59.055309+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6566,"log_dups_size":0,"ondisk_log_size":6566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'9771","reported_seq":15653,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:59.055894+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:18:59.055894+0000","last_peered":"2023-10-14T01:18:59.055894+0000","last_clean":"2023-10-14T01:18:59.055894+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:18:59.055894+0000","last_undegraded":"2023-10-14T01:18:59.055894+0000","last_fullsized":"2023-10-14T01:18:59.055894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9771,"log_dups_size":0,"ondisk_log_size":9771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"124'6540","reported_seq":9477,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:59.054920+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:18:59.054920+0000","last_peered":"2023-10-14T01:18:59.054920+0000","last_clean":"2023-10-14T01:18:59.054920+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:18:59.054920+0000","last_undegraded":"2023-10-14T01:18:59.054920+0000","last_fullsized":"2023-10-14T01:18:59.054920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'16117","reported_seq":23923,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:59.055485+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:18:59.055485+0000","last_peered":"2023-10-14T01:18:59.055485+0000","last_clean":"2023-10-14T01:18:59.055485+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:18:59.055485+0000","last_undegraded":"2023-10-14T01:18:59.055485+0000","last_fullsized":"2023-10-14T01:18:59.055485+0000","mapping_epoch":12,"log_start":"87'6100","ondisk_log_start":"87'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'512140","reported_seq":1018246,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:55.089291+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:18:55.089291+0000","last_peered":"2023-10-14T01:18:55.089291+0000","last_clean":"2023-10-14T01:18:55.089291+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:18:55.089291+0000","last_undegraded":"2023-10-14T01:18:55.089291+0000","last_fullsized":"2023-10-14T01:18:55.089291+0000","mapping_epoch":15,"log_start":"123'502136","ondisk_log_start":"123'502136","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2203,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 429s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7093125120,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":508196,"num_read_kb":2001564,"num_write":508104,"num_write_kb":2032416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"124'3397","reported_seq":5049,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:53.935591+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:18:53.935591+0000","last_peered":"2023-10-14T01:18:53.935591+0000","last_clean":"2023-10-14T01:18:53.935591+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:18:53.935591+0000","last_undegraded":"2023-10-14T01:18:53.935591+0000","last_fullsized":"2023-10-14T01:18:53.935591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'485374","reported_seq":966756,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:59.335567+0000","last_change":"2023-10-14T01:18:56.019468+0000","last_active":"2023-10-14T01:18:59.335567+0000","last_peered":"2023-10-14T01:18:59.335567+0000","last_clean":"2023-10-14T01:18:59.335567+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:18:59.335567+0000","last_undegraded":"2023-10-14T01:18:59.335567+0000","last_fullsized":"2023-10-14T01:18:59.335567+0000","mapping_epoch":15,"log_start":"123'475323","ondisk_log_start":"123'475323","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6738214912,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":483442,"num_read_kb":1903560,"num_write":482978,"num_write_kb":1931912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2979,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:53.940045+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:18:53.940045+0000","last_peered":"2023-10-14T01:18:53.940045+0000","last_clean":"2023-10-14T01:18:53.940045+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:18:53.940045+0000","last_undegraded":"2023-10-14T01:18:53.940045+0000","last_fullsized":"2023-10-14T01:18:53.940045+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'505490","reported_seq":1005148,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:55.045343+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:18:55.045343+0000","last_peered":"2023-10-14T01:18:55.045343+0000","last_clean":"2023-10-14T01:18:55.045343+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:18:55.045343+0000","last_undegraded":"2023-10-14T01:18:55.045343+0000","last_fullsized":"2023-10-14T01:18:55.045343+0000","mapping_epoch":15,"log_start":"123'495428","ondisk_log_start":"123'495428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2466,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 430s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6986240000,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":501671,"num_read_kb":1975684,"num_write":501324,"num_write_kb":2005296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3129,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:53.935147+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:18:53.935147+0000","last_peered":"2023-10-14T01:18:53.935147+0000","last_clean":"2023-10-14T01:18:53.935147+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:18:53.935147+0000","last_undegraded":"2023-10-14T01:18:53.935147+0000","last_fullsized":"2023-10-14T01:18:53.935147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'481026","reported_seq":956450,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:59.318685+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:18:59.318685+0000","last_peered":"2023-10-14T01:18:59.318685+0000","last_clean":"2023-10-14T01:18:59.318685+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:18:59.318685+0000","last_undegraded":"2023-10-14T01:18:59.318685+0000","last_fullsized":"2023-10-14T01:18:59.318685+0000","mapping_epoch":15,"log_start":"123'471025","ondisk_log_start":"123'471025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2586,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 413s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6644629523,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":477315,"num_read_kb":1879000,"num_write":476903,"num_write_kb":1907606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"124'9720","reported_seq":14019,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:18:59.055258+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:18:59.055258+0000","last_peered":"2023-10-14T01:18:59.055258+0000","last_clean":"2023-10-14T01:18:59.055258+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:18:59.055258+0000","last_undegraded":"2023-10-14T01:18:59.055258+0000","last_fullsized":"2023-10-14T01:18:59.055258+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9720,"log_dups_size":0,"ondisk_log_size":9720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":298,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-14T01:17:38.949301+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:17:38.949301+0000","last_peered":"2023-10-14T01:17:38.949301+0000","last_clean":"2023-10-14T01:17:38.949301+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:17:38.949301+0000","last_undegraded":"2023-10-14T01:17:38.949301+0000","last_fullsized":"2023-10-14T01:17:38.949301+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27462209555,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1970624,"num_read_kb":7759808,"num_write":1969309,"num_write_kb":7877230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16135155712,"data_stored":25317519398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40118,"ondisk_log_size":40118,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46235,"ondisk_log_size":46235,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739257,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8681824,"kb_used_data":7880856,"kb_used_omap":1270,"kb_used_meta":799689,"kb_avail":85690016,"statfs":{"total":96636764160,"available":87746576384,"internally_reserved":0,"allocated":8069996544,"data_stored":12660364270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":818882180},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.42099999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":2.5840000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.47199999999999998,"15min":0.55100000000000005},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.6419999999999999,"5min":4.1509999999999998,"15min":27.538},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:18:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.56100000000000005,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.16700000000000001},"max":{"1min":15.845000000000001,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.159,"5min":0.627,"15min":0.49299999999999999},"min":{"1min":0.23999999999999999,"5min":0.22700000000000001,"15min":0.186},"max":{"1min":15.805999999999999,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739256,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5637084,"kb_used_data":3841172,"kb_used_omap":472,"kb_used_meta":1795431,"kb_avail":88734756,"statfs":{"total":96636764160,"available":90864390144,"internally_reserved":0,"allocated":3933360128,"data_stored":6173802610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1838522063},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:49 2023","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":0.90700000000000003,"15min":0.60799999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":24.248999999999999,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":0.70599999999999996,"15min":0.60099999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":17.263999999999999,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:18:49 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.56299999999999994,"15min":0.53200000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.182},"max":{"1min":5.4790000000000001,"5min":5.4790000000000001,"15min":15.891},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.3149999999999999,"5min":0.84899999999999998,"15min":0.62},"min":{"1min":0.26200000000000001,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":22.498999999999999,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5887240,"kb_used_data":4038848,"kb_used_omap":500,"kb_used_meta":1847883,"kb_avail":88484600,"statfs":{"total":96636764160,"available":90608230400,"internally_reserved":0,"allocated":4135780352,"data_stored":6484625423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1892232805},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,6,6,18,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.16600000000000001},"max":{"1min":4.867,"5min":4.867,"15min":5.7729999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":3.1099999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.215}]},{"osd":2,"last update":"Sat Oct 14 01:18:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.47599999999999998},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.16300000000000001},"max":{"1min":1.635,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.317},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.249,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":0.89400000000000002,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.29999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4134629376,"data_stored":6484357120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3931672576,"data_stored":6173126675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8068853760,"data_stored":12660035603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:18:59.826 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:59.827 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:59.827 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:59.828 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:59.828 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:08.949 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:19:08.944+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:19:19.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:20.394 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:19:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:19:20.426 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2348,"stamp":"2023-10-14T01:19:19.503224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27491450597,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2011249,"num_read_kb":7816046,"num_write":1977452,"num_write_kb":7908867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86670,"ondisk_log_size":86670,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20405228,"kb_used_data":15819732,"kb_used_omap":2243,"kb_used_meta":4583228,"kb_avail":262710292,"statfs":{"total":289910292480,"available":269015339008,"internally_reserved":0,"allocated":16199405568,"data_stored":25379037420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2296904,"internal_metadata":4693226424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,9,7,11,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17981440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4333,"num_read_kb":17276,"num_write":4600,"num_write_kb":18400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002027"},"pg_stats":[{"pgid":"2.7","version":"125'6584","reported_seq":9729,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:17.654759+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:19:17.654759+0000","last_peered":"2023-10-14T01:19:17.654759+0000","last_clean":"2023-10-14T01:19:17.654759+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:19:17.654759+0000","last_undegraded":"2023-10-14T01:19:17.654759+0000","last_fullsized":"2023-10-14T01:19:17.654759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6584,"log_dups_size":0,"ondisk_log_size":6584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"125'9798","reported_seq":15691,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:17.666936+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:19:17.666936+0000","last_peered":"2023-10-14T01:19:17.666936+0000","last_clean":"2023-10-14T01:19:17.666936+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:19:17.666936+0000","last_undegraded":"2023-10-14T01:19:17.666936+0000","last_fullsized":"2023-10-14T01:19:17.666936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"125'6558","reported_seq":9502,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:17.652430+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:19:17.652430+0000","last_peered":"2023-10-14T01:19:17.652430+0000","last_clean":"2023-10-14T01:19:17.652430+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:19:17.652430+0000","last_undegraded":"2023-10-14T01:19:17.652430+0000","last_fullsized":"2023-10-14T01:19:17.652430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6558,"log_dups_size":0,"ondisk_log_size":6558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"125'16162","reported_seq":23984,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:17.689614+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:19:17.689614+0000","last_peered":"2023-10-14T01:19:17.689614+0000","last_clean":"2023-10-14T01:19:17.689614+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:19:17.689614+0000","last_undegraded":"2023-10-14T01:19:17.689614+0000","last_fullsized":"2023-10-14T01:19:17.689614+0000","mapping_epoch":12,"log_start":"87'6100","ondisk_log_start":"87'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"125'514154","reported_seq":1022040,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:15.196366+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:19:15.196366+0000","last_peered":"2023-10-14T01:19:15.196366+0000","last_clean":"2023-10-14T01:19:15.196366+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:19:15.196366+0000","last_undegraded":"2023-10-14T01:19:15.196366+0000","last_fullsized":"2023-10-14T01:19:15.196366+0000","mapping_epoch":15,"log_start":"123'504136","ondisk_log_start":"123'504136","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2363,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 450s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7100514304,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":509964,"num_read_kb":2008624,"num_write":510006,"num_write_kb":2040024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"125'3416","reported_seq":5073,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:12.942275+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:19:12.942275+0000","last_peered":"2023-10-14T01:19:12.942275+0000","last_clean":"2023-10-14T01:19:12.942275+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:19:12.942275+0000","last_undegraded":"2023-10-14T01:19:12.942275+0000","last_fullsized":"2023-10-14T01:19:12.942275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3416,"log_dups_size":0,"ondisk_log_size":3416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"125'487168","reported_seq":970257,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:19.370121+0000","last_change":"2023-10-14T01:19:19.056808+0000","last_active":"2023-10-14T01:19:19.370121+0000","last_peered":"2023-10-14T01:19:19.370121+0000","last_clean":"2023-10-14T01:19:19.370121+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:19:19.370121+0000","last_undegraded":"2023-10-14T01:19:19.370121+0000","last_fullsized":"2023-10-14T01:19:19.370121+0000","mapping_epoch":15,"log_start":"123'477123","ondisk_log_start":"123'477123","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6745247744,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":485141,"num_read_kb":1910340,"num_write":484772,"num_write_kb":1939088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2988,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.432582+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:19:14.432582+0000","last_peered":"2023-10-14T01:19:14.432582+0000","last_clean":"2023-10-14T01:19:14.432582+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:19:14.432582+0000","last_undegraded":"2023-10-14T01:19:14.432582+0000","last_fullsized":"2023-10-14T01:19:14.432582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"125'507487","reported_seq":1008931,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:15.192557+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:19:15.192557+0000","last_peered":"2023-10-14T01:19:15.192557+0000","last_clean":"2023-10-14T01:19:15.192557+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:19:15.192557+0000","last_undegraded":"2023-10-14T01:19:15.192557+0000","last_fullsized":"2023-10-14T01:19:15.192557+0000","mapping_epoch":15,"log_start":"123'497428","ondisk_log_start":"123'497428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2617,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 451s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":6993616896,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":503439,"num_read_kb":1982736,"num_write":503216,"num_write_kb":2012864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3138,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.416504+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:19:14.416504+0000","last_peered":"2023-10-14T01:19:14.416504+0000","last_clean":"2023-10-14T01:19:14.416504+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:19:14.416504+0000","last_undegraded":"2023-10-14T01:19:14.416504+0000","last_fullsized":"2023-10-14T01:19:14.416504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"125'482885","reported_seq":959967,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:19.414035+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:19:19.414035+0000","last_peered":"2023-10-14T01:19:19.414035+0000","last_clean":"2023-10-14T01:19:19.414035+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:19:19.414035+0000","last_undegraded":"2023-10-14T01:19:19.414035+0000","last_fullsized":"2023-10-14T01:19:19.414035+0000","mapping_epoch":15,"log_start":"123'472825","ondisk_log_start":"123'472825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2727,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 433s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6651478035,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":478957,"num_read_kb":1885540,"num_write":478662,"num_write_kb":1914642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"125'9747","reported_seq":14056,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:17.704238+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:19:17.704238+0000","last_peered":"2023-10-14T01:19:17.704238+0000","last_clean":"2023-10-14T01:19:17.704238+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:19:17.704238+0000","last_undegraded":"2023-10-14T01:19:17.704238+0000","last_fullsized":"2023-10-14T01:19:17.704238+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":299,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T01:17:41.298638+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:17:41.298638+0000","last_peered":"2023-10-14T01:17:41.298638+0000","last_clean":"2023-10-14T01:17:41.298638+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:17:41.298638+0000","last_undegraded":"2023-10-14T01:17:41.298638+0000","last_fullsized":"2023-10-14T01:17:41.298638+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27490856979,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1977501,"num_read_kb":7787240,"num_write":1976656,"num_write_kb":7906618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16195715072,"data_stored":25378078758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40182,"ondisk_log_size":40182,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46389,"ondisk_log_size":46389,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739261,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8801308,"kb_used_data":7910100,"kb_used_omap":1270,"kb_used_meta":889929,"kb_avail":85570532,"statfs":{"total":96636764160,"available":87624224768,"internally_reserved":0,"allocated":8099942400,"data_stored":12690302317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":911287940},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.42099999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":2.5840000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.47199999999999998,"15min":0.55100000000000005},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.6419999999999999,"5min":4.1509999999999998,"15min":27.538},"last":0.25600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:18:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.56100000000000005,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.16700000000000001},"max":{"1min":15.845000000000001,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.159,"5min":0.627,"15min":0.49299999999999999},"min":{"1min":0.23999999999999999,"5min":0.22700000000000001,"15min":0.186},"max":{"1min":15.805999999999999,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5665368,"kb_used_data":3855568,"kb_used_omap":472,"kb_used_meta":1809319,"kb_avail":88706472,"statfs":{"total":96636764160,"available":90835427328,"internally_reserved":0,"allocated":3948101632,"data_stored":6188536305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1852743375},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:49 2023","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":0.90700000000000003,"15min":0.60799999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":24.248999999999999,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.6539999999999999,"5min":0.70599999999999996,"15min":0.60099999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":17.263999999999999,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.373}]},{"osd":2,"last update":"Sat Oct 14 01:18:49 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.56299999999999994,"15min":0.53200000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.182},"max":{"1min":5.4790000000000001,"5min":5.4790000000000001,"15min":15.891},"last":0.224},{"interface":"front","average":{"1min":2.3149999999999999,"5min":0.84899999999999998,"15min":0.62},"min":{"1min":0.26200000000000001,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":22.498999999999999,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.40300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5938552,"kb_used_data":4054064,"kb_used_omap":500,"kb_used_meta":1883979,"kb_avail":88433288,"statfs":{"total":96636764160,"available":90555686912,"internally_reserved":0,"allocated":4151361536,"data_stored":6500198798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1929195109},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,7,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.16600000000000001},"max":{"1min":4.867,"5min":4.867,"15min":5.7729999999999997},"last":0.433},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":3.1099999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:18:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.47599999999999998},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.16300000000000001},"max":{"1min":1.635,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":2.7799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.249,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":0.89400000000000002,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":2.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4150173696,"data_stored":6499901440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3946242048,"data_stored":6187696147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8099299328,"data_stored":12690481171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:19:20.428 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:20.428 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:20.429 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:20.429 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:20.429 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:37.418 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:19:37.415+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:19:37.446 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:19:37.443+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:19:37.447 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:19:37.443+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:19:37.447 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:19:37.443+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:19:40.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:40.979 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:19:40.981 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:19:41.031 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2358,"stamp":"2023-10-14T01:19:39.506517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27521756901,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2018819,"num_read_kb":7846266,"num_write":1985236,"num_write_kb":7940003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86828,"ondisk_log_size":86828,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20497596,"kb_used_data":15881892,"kb_used_omap":2243,"kb_used_meta":4613436,"kb_avail":262617924,"statfs":{"total":289910292480,"available":268920754176,"internally_reserved":0,"allocated":16263057408,"data_stored":25442689260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2296904,"internal_metadata":4724159416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,8,8,10,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18837504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4719,"num_read_kb":18840,"num_write":4836,"num_write_kb":19344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001933"},"pg_stats":[{"pgid":"2.7","version":"125'6607","reported_seq":9760,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:38.303184+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:19:38.303184+0000","last_peered":"2023-10-14T01:19:38.303184+0000","last_clean":"2023-10-14T01:19:38.303184+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:19:38.303184+0000","last_undegraded":"2023-10-14T01:19:38.303184+0000","last_fullsized":"2023-10-14T01:19:38.303184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"125'9833","reported_seq":15738,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:38.319503+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:19:38.319503+0000","last_peered":"2023-10-14T01:19:38.319503+0000","last_clean":"2023-10-14T01:19:38.319503+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:19:38.319503+0000","last_undegraded":"2023-10-14T01:19:38.319503+0000","last_fullsized":"2023-10-14T01:19:38.319503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9833,"log_dups_size":0,"ondisk_log_size":9833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"125'6581","reported_seq":9533,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:38.306710+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:19:38.306710+0000","last_peered":"2023-10-14T01:19:38.306710+0000","last_clean":"2023-10-14T01:19:38.306710+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:19:38.306710+0000","last_undegraded":"2023-10-14T01:19:38.306710+0000","last_fullsized":"2023-10-14T01:19:38.306710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6581,"log_dups_size":0,"ondisk_log_size":6581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"125'16218","reported_seq":24060,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:38.328133+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:19:38.328133+0000","last_peered":"2023-10-14T01:19:38.328133+0000","last_clean":"2023-10-14T01:19:38.328133+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:19:38.328133+0000","last_undegraded":"2023-10-14T01:19:38.328133+0000","last_fullsized":"2023-10-14T01:19:38.328133+0000","mapping_epoch":12,"log_start":"89'6200","ondisk_log_start":"89'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"125'516282","reported_seq":1026184,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:35.373603+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:19:35.373603+0000","last_peered":"2023-10-14T01:19:35.373603+0000","last_clean":"2023-10-14T01:19:35.373603+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:19:35.373603+0000","last_undegraded":"2023-10-14T01:19:35.373603+0000","last_fullsized":"2023-10-14T01:19:35.373603+0000","mapping_epoch":15,"log_start":"124'506236","ondisk_log_start":"124'506236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2455,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 470s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7108567040,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":511969,"num_read_kb":2016636,"num_write":512071,"num_write_kb":2048284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"125'3427","reported_seq":5088,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:31.627343+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:19:31.627343+0000","last_peered":"2023-10-14T01:19:31.627343+0000","last_clean":"2023-10-14T01:19:31.627343+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:19:31.627343+0000","last_undegraded":"2023-10-14T01:19:31.627343+0000","last_fullsized":"2023-10-14T01:19:31.627343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"125'489060","reported_seq":974001,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:39.496615+0000","last_change":"2023-10-14T01:19:36.362857+0000","last_active":"2023-10-14T01:19:39.496615+0000","last_peered":"2023-10-14T01:19:39.496615+0000","last_clean":"2023-10-14T01:19:39.496615+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:19:39.496615+0000","last_undegraded":"2023-10-14T01:19:39.496615+0000","last_fullsized":"2023-10-14T01:19:39.496615+0000","mapping_epoch":15,"log_start":"124'479023","ondisk_log_start":"124'479023","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6752698368,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":486988,"num_read_kb":1917720,"num_write":486664,"num_write_kb":1946656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":2996,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:34.901942+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:19:34.901942+0000","last_peered":"2023-10-14T01:19:34.901942+0000","last_clean":"2023-10-14T01:19:34.901942+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:19:34.901942+0000","last_undegraded":"2023-10-14T01:19:34.901942+0000","last_fullsized":"2023-10-14T01:19:34.901942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"125'509516","reported_seq":1012849,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:35.365655+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:19:35.365655+0000","last_peered":"2023-10-14T01:19:35.365655+0000","last_clean":"2023-10-14T01:19:35.365655+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:19:35.365655+0000","last_undegraded":"2023-10-14T01:19:35.365655+0000","last_fullsized":"2023-10-14T01:19:35.365655+0000","mapping_epoch":15,"log_start":"124'499428","ondisk_log_start":"124'499428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2698,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 471s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7001255936,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":505318,"num_read_kb":1990232,"num_write":505187,"num_write_kb":2020748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3146,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:34.897317+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:19:34.897317+0000","last_peered":"2023-10-14T01:19:34.897317+0000","last_clean":"2023-10-14T01:19:34.897317+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:19:34.897317+0000","last_undegraded":"2023-10-14T01:19:34.897317+0000","last_fullsized":"2023-10-14T01:19:34.897317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"125'484813","reported_seq":963742,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:39.494075+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:19:39.494075+0000","last_peered":"2023-10-14T01:19:39.494075+0000","last_clean":"2023-10-14T01:19:39.494075+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:19:39.494075+0000","last_undegraded":"2023-10-14T01:19:39.494075+0000","last_fullsized":"2023-10-14T01:19:39.494075+0000","mapping_epoch":15,"log_start":"124'474725","ondisk_log_start":"124'474725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2829,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 453s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6658641939,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":480796,"num_read_kb":1892872,"num_write":480518,"num_write_kb":1922066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"125'9780","reported_seq":14101,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:38.329127+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:19:38.329127+0000","last_peered":"2023-10-14T01:19:38.329127+0000","last_clean":"2023-10-14T01:19:38.329127+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:19:38.329127+0000","last_undegraded":"2023-10-14T01:19:38.329127+0000","last_fullsized":"2023-10-14T01:19:38.329127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9780,"log_dups_size":0,"ondisk_log_size":9780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":300,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:01.305956+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:19:01.305956+0000","last_peered":"2023-10-14T01:19:01.305956+0000","last_clean":"2023-10-14T01:19:01.305956+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:19:01.305956+0000","last_undegraded":"2023-10-14T01:19:01.305956+0000","last_fullsized":"2023-10-14T01:19:01.305956+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27521163283,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1985071,"num_read_kb":7817460,"num_write":1984440,"num_write_kb":7937754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16259133440,"data_stored":25441497126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46470,"ondisk_log_size":46470,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739265,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8748032,"kb_used_data":7941304,"kb_used_omap":1270,"kb_used_meta":805449,"kb_avail":85623808,"statfs":{"total":96636764160,"available":87678779392,"internally_reserved":0,"allocated":8131895296,"data_stored":12722255213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":824780420},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.37,"15min":0.42099999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":2.5840000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.216},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.47199999999999998,"15min":0.55100000000000005},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.6419999999999999,"5min":4.1509999999999998,"15min":27.538},"last":0.32700000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:18:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.56100000000000005,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.16700000000000001},"max":{"1min":15.845000000000001,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.159,"5min":0.627,"15min":0.49299999999999999},"min":{"1min":0.23999999999999999,"5min":0.22700000000000001,"15min":0.186},"max":{"1min":15.805999999999999,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.36799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739264,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5758900,"kb_used_data":3870508,"kb_used_omap":472,"kb_used_meta":1887911,"kb_avail":88612940,"statfs":{"total":96636764160,"available":90739650560,"internally_reserved":0,"allocated":3963400192,"data_stored":6203834865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1933221583},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:49 2023","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":0.90700000000000003,"15min":0.60799999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":24.248999999999999,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.6539999999999999,"5min":0.70599999999999996,"15min":0.60099999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":17.263999999999999,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:18:49 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.56299999999999994,"15min":0.53200000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.182},"max":{"1min":5.4790000000000001,"5min":5.4790000000000001,"15min":15.891},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.3149999999999999,"5min":0.84899999999999998,"15min":0.62},"min":{"1min":0.26200000000000001,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":22.498999999999999,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739263,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5990664,"kb_used_data":4070080,"kb_used_omap":500,"kb_used_meta":1920075,"kb_avail":88381176,"statfs":{"total":96636764160,"available":90502324224,"internally_reserved":0,"allocated":4167761920,"data_stored":6516599182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":1966157413},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,3,6,5,5,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.16600000000000001},"max":{"1min":4.867,"5min":4.867,"15min":5.7729999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":3.1099999999999999,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.221}]},{"osd":2,"last update":"Sat Oct 14 01:18:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.53500000000000003,"15min":0.47599999999999998},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.16300000000000001},"max":{"1min":1.635,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.249,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":0.89400000000000002,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4166688768,"data_stored":6516416512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3961573376,"data_stored":6203027475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8130871296,"data_stored":12722053139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:19:41.033 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:41.034 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:41.034 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:41.035 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:41.035 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:01.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:01.557 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:20:01.558 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:20:01.615 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2369,"stamp":"2023-10-14T01:20:01.510200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27555340005,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2027512,"num_read_kb":7880978,"num_write":1993872,"num_write_kb":7974547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86883,"ondisk_log_size":86883,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20748164,"kb_used_data":15947948,"kb_used_omap":2243,"kb_used_meta":4797948,"kb_avail":262367356,"statfs":{"total":289910292480,"available":268664172544,"internally_reserved":0,"allocated":16330698752,"data_stored":25510330604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2296904,"internal_metadata":4913099704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,6,14,22,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22212608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5845,"num_read_kb":23344,"num_write":5725,"num_write_kb":22900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001971"},"pg_stats":[{"pgid":"2.7","version":"125'6629","reported_seq":9790,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:57.826950+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:19:57.826950+0000","last_peered":"2023-10-14T01:19:57.826950+0000","last_clean":"2023-10-14T01:19:57.826950+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:19:57.826950+0000","last_undegraded":"2023-10-14T01:19:57.826950+0000","last_fullsized":"2023-10-14T01:19:57.826950+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6629,"log_dups_size":0,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"125'9866","reported_seq":15783,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:57.837742+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:19:57.837742+0000","last_peered":"2023-10-14T01:19:57.837742+0000","last_clean":"2023-10-14T01:19:57.837742+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:19:57.837742+0000","last_undegraded":"2023-10-14T01:19:57.837742+0000","last_fullsized":"2023-10-14T01:19:57.837742+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9866,"log_dups_size":0,"ondisk_log_size":9866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"125'6603","reported_seq":9563,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:57.818688+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:19:57.818688+0000","last_peered":"2023-10-14T01:19:57.818688+0000","last_clean":"2023-10-14T01:19:57.818688+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:19:57.818688+0000","last_undegraded":"2023-10-14T01:19:57.818688+0000","last_fullsized":"2023-10-14T01:19:57.818688+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"125'16273","reported_seq":24135,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:57.855669+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:19:57.855669+0000","last_peered":"2023-10-14T01:19:57.855669+0000","last_clean":"2023-10-14T01:19:57.855669+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:19:57.855669+0000","last_undegraded":"2023-10-14T01:19:57.855669+0000","last_fullsized":"2023-10-14T01:19:57.855669+0000","mapping_epoch":12,"log_start":"89'6200","ondisk_log_start":"89'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"125'518875","reported_seq":1031329,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:00.537121+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:20:00.537121+0000","last_peered":"2023-10-14T01:20:00.537121+0000","last_clean":"2023-10-14T01:20:00.537121+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:20:00.537121+0000","last_undegraded":"2023-10-14T01:20:00.537121+0000","last_fullsized":"2023-10-14T01:20:00.537121+0000","mapping_epoch":15,"log_start":"124'508836","ondisk_log_start":"124'508836","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2585,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 495s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7118327808,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":514507,"num_read_kb":2026764,"num_write":514578,"num_write_kb":2058312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"125'3439","reported_seq":5105,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:59.305142+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:19:59.305142+0000","last_peered":"2023-10-14T01:19:59.305142+0000","last_clean":"2023-10-14T01:19:59.305142+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:19:59.305142+0000","last_undegraded":"2023-10-14T01:19:59.305142+0000","last_fullsized":"2023-10-14T01:19:59.305142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"125'490953","reported_seq":977801,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:59.583929+0000","last_change":"2023-10-14T01:19:58.181449+0000","last_active":"2023-10-14T01:19:59.583929+0000","last_peered":"2023-10-14T01:19:59.583929+0000","last_clean":"2023-10-14T01:19:59.583929+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:19:59.583929+0000","last_undegraded":"2023-10-14T01:19:59.583929+0000","last_fullsized":"2023-10-14T01:19:59.583929+0000","mapping_epoch":15,"log_start":"124'480923","ondisk_log_start":"124'480923","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6760108032,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":488887,"num_read_kb":1925292,"num_write":488557,"num_write_kb":1954228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3004,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:55.382772+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:19:55.382772+0000","last_peered":"2023-10-14T01:19:55.382772+0000","last_clean":"2023-10-14T01:19:55.382772+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:19:55.382772+0000","last_undegraded":"2023-10-14T01:19:55.382772+0000","last_fullsized":"2023-10-14T01:19:55.382772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"125'511976","reported_seq":1017712,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:00.508715+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:20:00.508715+0000","last_peered":"2023-10-14T01:20:00.508715+0000","last_clean":"2023-10-14T01:20:00.508715+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:20:00.508715+0000","last_undegraded":"2023-10-14T01:20:00.508715+0000","last_fullsized":"2023-10-14T01:20:00.508715+0000","mapping_epoch":15,"log_start":"124'501928","ondisk_log_start":"124'501928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2829,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 496s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7010385920,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":507706,"num_read_kb":1999776,"num_write":507553,"num_write_kb":2030212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3155,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:55.375394+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:19:55.375394+0000","last_peered":"2023-10-14T01:19:55.375394+0000","last_clean":"2023-10-14T01:19:55.375394+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:19:55.375394+0000","last_undegraded":"2023-10-14T01:19:55.375394+0000","last_fullsized":"2023-10-14T01:19:55.375394+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"125'486745","reported_seq":967561,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:59.600671+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:19:59.600671+0000","last_peered":"2023-10-14T01:19:59.600671+0000","last_clean":"2023-10-14T01:19:59.600671+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:19:59.600671+0000","last_undegraded":"2023-10-14T01:19:59.600671+0000","last_fullsized":"2023-10-14T01:19:59.600671+0000","mapping_epoch":15,"log_start":"124'476725","ondisk_log_start":"124'476725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2921,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 474s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6665924627,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":482664,"num_read_kb":1900340,"num_write":482388,"num_write_kb":1929546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"125'9813","reported_seq":14146,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:57.828504+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:19:57.828504+0000","last_peered":"2023-10-14T01:19:57.828504+0000","last_clean":"2023-10-14T01:19:57.828504+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:19:57.828504+0000","last_undegraded":"2023-10-14T01:19:57.828504+0000","last_fullsized":"2023-10-14T01:19:57.828504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":300,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:01.305956+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:19:01.305956+0000","last_peered":"2023-10-14T01:19:01.305956+0000","last_clean":"2023-10-14T01:19:01.305956+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:19:01.305956+0000","last_undegraded":"2023-10-14T01:19:01.305956+0000","last_fullsized":"2023-10-14T01:19:01.305956+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27554746387,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":1993764,"num_read_kb":7852172,"num_write":1993076,"num_write_kb":7972298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16325898240,"data_stored":25508261926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40137,"ondisk_log_size":40137,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46647,"ondisk_log_size":46647,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739269,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8885520,"kb_used_data":7971976,"kb_used_omap":1270,"kb_used_meta":912265,"kb_avail":85486320,"statfs":{"total":96636764160,"available":87537991680,"internally_reserved":0,"allocated":8163303424,"data_stored":12753663341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":934160004},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.379,"15min":0.42699999999999999},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":2.3980000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.42499999999999999,"15min":0.53100000000000003},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56599999999999995,"5min":4.1509999999999998,"15min":27.538},"last":0.37}]},{"osd":1,"last update":"Sat Oct 14 01:19:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59799999999999998,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23300000000000001,"15min":0.16700000000000001},"max":{"1min":3.7509999999999999,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.65300000000000002,"15min":0.504},"min":{"1min":0.22,"5min":0.22,"15min":0.186},"max":{"1min":3.802,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359739268,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5809984,"kb_used_data":3885560,"kb_used_omap":472,"kb_used_meta":1923943,"kb_avail":88561856,"statfs":{"total":96636764160,"available":90687340544,"internally_reserved":0,"allocated":3978813440,"data_stored":6219248113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":1970118351},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.86699999999999999,"15min":0.58899999999999997},"min":{"1min":0.216,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":0.76600000000000001,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.67300000000000004,"15min":0.59699999999999998},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.92200000000000004,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.373}]},{"osd":2,"last update":"Sat Oct 14 01:19:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.57599999999999996,"15min":0.50600000000000001},"min":{"1min":0.216,"5min":0.185,"15min":0.182},"max":{"1min":1.196,"5min":5.4790000000000001,"15min":15.891},"last":0.25},{"interface":"front","average":{"1min":0.372,"5min":0.76500000000000001,"15min":0.59899999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.627,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6052660,"kb_used_data":4090412,"kb_used_omap":500,"kb_used_meta":1961739,"kb_avail":88319180,"statfs":{"total":96636764160,"available":90438840320,"internally_reserved":0,"allocated":4188581888,"data_stored":6537419150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":2008821349},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,7,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.46700000000000003},"min":{"1min":0.22700000000000001,"5min":0.185,"15min":0.16600000000000001},"max":{"1min":2.0379999999999998,"5min":4.867,"15min":5.7729999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.315,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.21199999999999999,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":0.58499999999999996,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.23300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.52300000000000002,"15min":0.48499999999999999},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.16300000000000001},"max":{"1min":2.7799999999999998,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.47599999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.7679999999999998,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4186656768,"data_stored":6536384512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3976998912,"data_stored":6218453011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8162242560,"data_stored":12753424403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:20:01.617 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:01.617 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:01.617 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:01.618 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:01.618 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:07.007 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:20:07.003+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:20:20.630 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:20:20.618+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:20:21.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:22.153 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:20:22.154 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:20:22.184 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2380,"stamp":"2023-10-14T01:20:21.513470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27585085157,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2035191,"num_read_kb":7911622,"num_write":2001456,"num_write_kb":8004883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86937,"ondisk_log_size":86937,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21001176,"kb_used_data":16008320,"kb_used_omap":2243,"kb_used_meta":4990588,"kb_avail":262114344,"statfs":{"total":289910292480,"available":268405088256,"internally_reserved":0,"allocated":16392519680,"data_stored":25572127976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2296904,"internal_metadata":5110363064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,7,13,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14860288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3943,"num_read_kb":15748,"num_write":3757,"num_write_kb":15028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.201420"},"pg_stats":[{"pgid":"2.7","version":"126'6645","reported_seq":9815,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:18.416092+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:20:18.416092+0000","last_peered":"2023-10-14T01:20:18.416092+0000","last_clean":"2023-10-14T01:20:18.416092+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:20:18.416092+0000","last_undegraded":"2023-10-14T01:20:18.416092+0000","last_fullsized":"2023-10-14T01:20:18.416092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'9890","reported_seq":15820,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:18.439202+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:20:18.439202+0000","last_peered":"2023-10-14T01:20:18.439202+0000","last_clean":"2023-10-14T01:20:18.439202+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:20:18.439202+0000","last_undegraded":"2023-10-14T01:20:18.439202+0000","last_fullsized":"2023-10-14T01:20:18.439202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"126'6619","reported_seq":9588,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:18.416258+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:20:18.416258+0000","last_peered":"2023-10-14T01:20:18.416258+0000","last_clean":"2023-10-14T01:20:18.416258+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:20:18.416258+0000","last_undegraded":"2023-10-14T01:20:18.416258+0000","last_fullsized":"2023-10-14T01:20:18.416258+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'16313","reported_seq":24196,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:18.462897+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:20:18.462897+0000","last_peered":"2023-10-14T01:20:18.462897+0000","last_clean":"2023-10-14T01:20:18.462897+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:20:18.462897+0000","last_undegraded":"2023-10-14T01:20:18.462897+0000","last_fullsized":"2023-10-14T01:20:18.462897+0000","mapping_epoch":12,"log_start":"89'6300","ondisk_log_start":"89'6300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'520849","reported_seq":1035399,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:20.613918+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:20:20.613918+0000","last_peered":"2023-10-14T01:20:20.613918+0000","last_clean":"2023-10-14T01:20:20.613918+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:20:20.613918+0000","last_undegraded":"2023-10-14T01:20:20.613918+0000","last_fullsized":"2023-10-14T01:20:20.613918+0000","mapping_epoch":15,"log_start":"124'510836","ondisk_log_start":"124'510836","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2694,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7125790720,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":516586,"num_read_kb":2035064,"num_write":516477,"num_write_kb":2065908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"126'3451","reported_seq":5121,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:20.576929+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:20:20.576929+0000","last_peered":"2023-10-14T01:20:20.576929+0000","last_clean":"2023-10-14T01:20:20.576929+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:20:20.576929+0000","last_undegraded":"2023-10-14T01:20:20.576929+0000","last_fullsized":"2023-10-14T01:20:20.576929+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'492850","reported_seq":981576,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:19.715544+0000","last_change":"2023-10-14T01:20:18.216279+0000","last_active":"2023-10-14T01:20:19.715544+0000","last_peered":"2023-10-14T01:20:19.715544+0000","last_clean":"2023-10-14T01:20:19.715544+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:20:19.715544+0000","last_undegraded":"2023-10-14T01:20:19.715544+0000","last_fullsized":"2023-10-14T01:20:19.715544+0000","mapping_epoch":15,"log_start":"124'482823","ondisk_log_start":"124'482823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6767640576,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":490755,"num_read_kb":1932760,"num_write":490454,"num_write_kb":1961816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3014,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:15.952466+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:20:15.952466+0000","last_peered":"2023-10-14T01:20:15.952466+0000","last_clean":"2023-10-14T01:20:15.952466+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:20:15.952466+0000","last_undegraded":"2023-10-14T01:20:15.952466+0000","last_fullsized":"2023-10-14T01:20:15.952466+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'513974","reported_seq":1021695,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:20.591055+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:20:20.591055+0000","last_peered":"2023-10-14T01:20:20.591055+0000","last_clean":"2023-10-14T01:20:20.591055+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:20:20.591055+0000","last_undegraded":"2023-10-14T01:20:20.591055+0000","last_fullsized":"2023-10-14T01:20:20.591055+0000","mapping_epoch":15,"log_start":"124'503928","ondisk_log_start":"124'503928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2925,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 516s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7017852928,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":509677,"num_read_kb":2007632,"num_write":509485,"num_write_kb":2037940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3164,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:15.858549+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:20:15.858549+0000","last_peered":"2023-10-14T01:20:15.858549+0000","last_clean":"2023-10-14T01:20:15.858549+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:20:15.858549+0000","last_undegraded":"2023-10-14T01:20:15.858549+0000","last_fullsized":"2023-10-14T01:20:15.858549+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'488699","reported_seq":971285,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:19.734320+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:20:19.734320+0000","last_peered":"2023-10-14T01:20:19.734320+0000","last_clean":"2023-10-14T01:20:19.734320+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:20:19.734320+0000","last_undegraded":"2023-10-14T01:20:19.734320+0000","last_fullsized":"2023-10-14T01:20:19.734320+0000","mapping_epoch":15,"log_start":"124'478626","ondisk_log_start":"124'478626","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3065,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 494s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6673207315,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":484425,"num_read_kb":1907360,"num_write":484244,"num_write_kb":1936970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"126'9837","reported_seq":14183,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:18.460141+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:20:18.460141+0000","last_peered":"2023-10-14T01:20:18.460141+0000","last_clean":"2023-10-14T01:20:18.460141+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:20:18.460141+0000","last_undegraded":"2023-10-14T01:20:18.460141+0000","last_fullsized":"2023-10-14T01:20:18.460141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9837,"log_dups_size":0,"ondisk_log_size":9837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":301,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:39.503510+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:19:39.503510+0000","last_peered":"2023-10-14T01:19:39.503510+0000","last_clean":"2023-10-14T01:19:39.503510+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:19:39.503510+0000","last_undegraded":"2023-10-14T01:19:39.503510+0000","last_fullsized":"2023-10-14T01:19:39.503510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27584491539,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2001443,"num_read_kb":7882816,"num_write":2000660,"num_write_kb":8002634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16388063232,"data_stored":25570426918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40159,"ondisk_log_size":40159,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46679,"ondisk_log_size":46679,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739273,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9029756,"kb_used_data":8002100,"kb_used_omap":1270,"kb_used_meta":1026377,"kb_avail":85342084,"statfs":{"total":96636764160,"available":87390294016,"internally_reserved":0,"allocated":8194150400,"data_stored":12784502465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1051010692},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.379,"15min":0.42699999999999999},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":2.3980000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.42499999999999999,"15min":0.53100000000000003},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56599999999999995,"5min":4.1509999999999998,"15min":27.538},"last":0.27700000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:19:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59799999999999998,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23300000000000001,"15min":0.16700000000000001},"max":{"1min":3.7509999999999999,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.313},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.65300000000000002,"15min":0.504},"min":{"1min":0.22,"5min":0.22,"15min":0.186},"max":{"1min":3.802,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.24399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739272,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5885360,"kb_used_data":3900456,"kb_used_omap":472,"kb_used_meta":1984423,"kb_avail":88486480,"statfs":{"total":96636764160,"available":90610155520,"internally_reserved":0,"allocated":3994066944,"data_stored":6234493765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":2032049871},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.86699999999999999,"15min":0.58899999999999997},"min":{"1min":0.216,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":0.76600000000000001,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.67300000000000004,"15min":0.59699999999999998},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.92200000000000004,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:19:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.57599999999999996,"15min":0.50600000000000001},"min":{"1min":0.216,"5min":0.185,"15min":0.182},"max":{"1min":1.196,"5min":5.4790000000000001,"15min":15.891},"last":0.25},{"interface":"front","average":{"1min":0.372,"5min":0.76500000000000001,"15min":0.59899999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.627,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.379}]}]},{"osd":0,"up_from":8,"seq":34359739272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6086060,"kb_used_data":4105764,"kb_used_omap":500,"kb_used_meta":1979787,"kb_avail":88285780,"statfs":{"total":96636764160,"available":90404638720,"internally_reserved":0,"allocated":4204302336,"data_stored":6553131746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":512411,"internal_metadata":2027302501},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,5,9,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.46700000000000003},"min":{"1min":0.22700000000000001,"5min":0.185,"15min":0.16600000000000001},"max":{"1min":2.0379999999999998,"5min":4.867,"15min":5.7729999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.21199999999999999,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":0.58499999999999996,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.86199999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.52300000000000002,"15min":0.48499999999999999},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.16300000000000001},"max":{"1min":2.7799999999999998,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.308},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.47599999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.7679999999999998,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4202340352,"data_stored":6552068096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3992375296,"data_stored":6233829395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8193347584,"data_stored":12784529427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:20:22.186 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:22.187 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:22.187 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:22.188 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:22.188 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:42.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:20:42.748 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:20:42.773 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2390,"stamp":"2023-10-14T01:20:41.516992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27614510821,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2042860,"num_read_kb":7942218,"num_write":2008963,"num_write_kb":8034911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87078,"ondisk_log_size":87078,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20654168,"kb_used_data":16068096,"kb_used_omap":2244,"kb_used_meta":4583803,"kb_avail":262461352,"statfs":{"total":289910292480,"available":268760424448,"internally_reserved":0,"allocated":16453730304,"data_stored":25633338600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2298584,"internal_metadata":4693814568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,1,8,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22196224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5707,"num_read_kb":22776,"num_write":5664,"num_write_kb":22656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002054"},"pg_stats":[{"pgid":"2.7","version":"126'6665","reported_seq":9843,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:37.119785+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:20:37.119785+0000","last_peered":"2023-10-14T01:20:37.119785+0000","last_clean":"2023-10-14T01:20:37.119785+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:20:37.119785+0000","last_undegraded":"2023-10-14T01:20:37.119785+0000","last_fullsized":"2023-10-14T01:20:37.119785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6665,"log_dups_size":0,"ondisk_log_size":6665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'9920","reported_seq":15862,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:37.134232+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:20:37.134232+0000","last_peered":"2023-10-14T01:20:37.134232+0000","last_clean":"2023-10-14T01:20:37.134232+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:20:37.134232+0000","last_undegraded":"2023-10-14T01:20:37.134232+0000","last_fullsized":"2023-10-14T01:20:37.134232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9920,"log_dups_size":0,"ondisk_log_size":9920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"126'6639","reported_seq":9616,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:37.128619+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:20:37.128619+0000","last_peered":"2023-10-14T01:20:37.128619+0000","last_clean":"2023-10-14T01:20:37.128619+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:20:37.128619+0000","last_undegraded":"2023-10-14T01:20:37.128619+0000","last_fullsized":"2023-10-14T01:20:37.128619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'16363","reported_seq":24266,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:37.161014+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:20:37.161014+0000","last_peered":"2023-10-14T01:20:37.161014+0000","last_clean":"2023-10-14T01:20:37.161014+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:20:37.161014+0000","last_undegraded":"2023-10-14T01:20:37.161014+0000","last_fullsized":"2023-10-14T01:20:37.161014+0000","mapping_epoch":12,"log_start":"89'6300","ondisk_log_start":"89'6300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'522844","reported_seq":1039338,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:40.689515+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:20:40.689515+0000","last_peered":"2023-10-14T01:20:40.689515+0000","last_clean":"2023-10-14T01:20:40.689515+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:20:40.689515+0000","last_undegraded":"2023-10-14T01:20:40.689515+0000","last_fullsized":"2023-10-14T01:20:40.689515+0000","mapping_epoch":15,"log_start":"125'512836","ondisk_log_start":"125'512836","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2826,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 535s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7133294592,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":518534,"num_read_kb":2042840,"num_write":518383,"num_write_kb":2073532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"126'3468","reported_seq":5142,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:39.337390+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:20:39.337390+0000","last_peered":"2023-10-14T01:20:39.337390+0000","last_clean":"2023-10-14T01:20:39.337390+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:20:39.337390+0000","last_undegraded":"2023-10-14T01:20:39.337390+0000","last_fullsized":"2023-10-14T01:20:39.337390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3468,"log_dups_size":0,"ondisk_log_size":3468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'494739","reported_seq":985374,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:39.814020+0000","last_change":"2023-10-14T01:20:38.337774+0000","last_active":"2023-10-14T01:20:39.814020+0000","last_peered":"2023-10-14T01:20:39.814020+0000","last_clean":"2023-10-14T01:20:39.814020+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:20:39.814020+0000","last_undegraded":"2023-10-14T01:20:39.814020+0000","last_fullsized":"2023-10-14T01:20:39.814020+0000","mapping_epoch":15,"log_start":"124'484723","ondisk_log_start":"124'484723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6775029760,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":492661,"num_read_kb":1940364,"num_write":492343,"num_write_kb":1969372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3022,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:36.339372+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:20:36.339372+0000","last_peered":"2023-10-14T01:20:36.339372+0000","last_clean":"2023-10-14T01:20:36.339372+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:20:36.339372+0000","last_undegraded":"2023-10-14T01:20:36.339372+0000","last_fullsized":"2023-10-14T01:20:36.339372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'516005","reported_seq":1025643,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:40.791465+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:20:40.791465+0000","last_peered":"2023-10-14T01:20:40.791465+0000","last_clean":"2023-10-14T01:20:40.791465+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:20:40.791465+0000","last_undegraded":"2023-10-14T01:20:40.791465+0000","last_fullsized":"2023-10-14T01:20:40.791465+0000","mapping_epoch":15,"log_start":"124'505928","ondisk_log_start":"124'505928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3045,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 536s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7025446912,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":511580,"num_read_kb":2015232,"num_write":511433,"num_write_kb":2045732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3172,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:36.340041+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:20:36.340041+0000","last_peered":"2023-10-14T01:20:36.340041+0000","last_clean":"2023-10-14T01:20:36.340041+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:20:36.340041+0000","last_undegraded":"2023-10-14T01:20:36.340041+0000","last_fullsized":"2023-10-14T01:20:36.340041+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'490559","reported_seq":975069,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:39.865496+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:20:39.865496+0000","last_peered":"2023-10-14T01:20:39.865496+0000","last_clean":"2023-10-14T01:20:39.865496+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:20:39.865496+0000","last_undegraded":"2023-10-14T01:20:39.865496+0000","last_fullsized":"2023-10-14T01:20:39.865496+0000","mapping_epoch":15,"log_start":"124'480527","ondisk_log_start":"124'480527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3202,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 514s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6680145939,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":486337,"num_read_kb":1914976,"num_write":486008,"num_write_kb":1944026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"126'9867","reported_seq":14225,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:37.186689+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:20:37.186689+0000","last_peered":"2023-10-14T01:20:37.186689+0000","last_clean":"2023-10-14T01:20:37.186689+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:20:37.186689+0000","last_undegraded":"2023-10-14T01:20:37.186689+0000","last_fullsized":"2023-10-14T01:20:37.186689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9867,"log_dups_size":0,"ondisk_log_size":9867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":301,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:39.503510+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:19:39.503510+0000","last_peered":"2023-10-14T01:19:39.503510+0000","last_clean":"2023-10-14T01:19:39.503510+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:19:39.503510+0000","last_undegraded":"2023-10-14T01:19:39.503510+0000","last_fullsized":"2023-10-14T01:19:39.503510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27613917203,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2009112,"num_read_kb":7913412,"num_write":2008167,"num_write_kb":8032662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16449474560,"data_stored":25631838246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40133,"ondisk_log_size":40133,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46846,"ondisk_log_size":46846,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739277,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9127532,"kb_used_data":8031844,"kb_used_omap":1270,"kb_used_meta":1094409,"kb_avail":85244308,"statfs":{"total":96636764160,"available":87290171392,"internally_reserved":0,"allocated":8224608256,"data_stored":12814960321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1120675460},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.379,"15min":0.42699999999999999},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":2.3980000000000001,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.182},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.42499999999999999,"15min":0.53100000000000003},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.56599999999999995,"5min":4.1509999999999998,"15min":27.538},"last":0.24399999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:19:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59799999999999998,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23300000000000001,"15min":0.16700000000000001},"max":{"1min":3.7509999999999999,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.65300000000000002,"15min":0.504},"min":{"1min":0.22,"5min":0.22,"15min":0.186},"max":{"1min":3.802,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.219}]}]},{"osd":1,"up_from":8,"seq":34359739276,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5948476,"kb_used_data":3915060,"kb_used_omap":472,"kb_used_meta":2032935,"kb_avail":88423364,"statfs":{"total":96636764160,"available":90545524736,"internally_reserved":0,"allocated":4009021440,"data_stored":6249448261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":2081726159},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.86699999999999999,"15min":0.58899999999999997},"min":{"1min":0.216,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":0.76600000000000001,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.67300000000000004,"15min":0.59699999999999998},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.92200000000000004,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:19:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.57599999999999996,"15min":0.50600000000000001},"min":{"1min":0.216,"5min":0.185,"15min":0.182},"max":{"1min":1.196,"5min":5.4790000000000001,"15min":15.891},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.76500000000000001,"15min":0.59899999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.627,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5578160,"kb_used_data":4121192,"kb_used_omap":502,"kb_used_meta":1456457,"kb_avail":88793680,"statfs":{"total":96636764160,"available":90924728320,"internally_reserved":0,"allocated":4220100608,"data_stored":6568930018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1491412949},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,1,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.46700000000000003},"min":{"1min":0.22700000000000001,"5min":0.185,"15min":0.16600000000000001},"max":{"1min":2.0379999999999998,"5min":4.867,"15min":5.7729999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.21199999999999999,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":0.58499999999999996,"5min":3.5739999999999998,"15min":5.0060000000000002},"last":0.29299999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.52300000000000002,"15min":0.48499999999999999},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.16300000000000001},"max":{"1min":2.7799999999999998,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.47599999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.7679999999999998,"5min":4.8049999999999997,"15min":4.9690000000000003},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4218134528,"data_stored":6567862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4007342080,"data_stored":6248796179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8223997952,"data_stored":12815179795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:20:42.775 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:42.775 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:42.776 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:42.776 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:42.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:20:43.280 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:20:43.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:20:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:20:43.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:20:44.499 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:20:44.532 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:20:45.075 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:20:45.127 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:00.042 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:21:00.042 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:21:00.044 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:00.044 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:21:00.044 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:00.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:21:00.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:21:00.045 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:21:00.046 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:21:00.046 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:21:00.046 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:00.047 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:00.047 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:21:00.047 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:21:00.048 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:00.042+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:21:05.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:05.649 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:21:05.650 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:21:05.676 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2402,"stamp":"2023-10-14T01:21:05.520955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27647639269,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2051621,"num_read_kb":7977110,"num_write":2017415,"num_write_kb":8068719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87368,"ondisk_log_size":87368,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20924712,"kb_used_data":16138704,"kb_used_omap":2244,"kb_used_meta":4783739,"kb_avail":262190808,"statfs":{"total":289910292480,"available":268483387392,"internally_reserved":0,"allocated":16526032896,"data_stored":25705641192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2298584,"internal_metadata":4898549032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,11,13,19,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18538496,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4792,"num_read_kb":19108,"num_write":4746,"num_write_kb":18984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001989"},"pg_stats":[{"pgid":"2.7","version":"126'6693","reported_seq":9881,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:04.532845+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:21:04.532845+0000","last_peered":"2023-10-14T01:21:04.532845+0000","last_clean":"2023-10-14T01:21:04.532845+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:21:04.532845+0000","last_undegraded":"2023-10-14T01:21:04.532845+0000","last_fullsized":"2023-10-14T01:21:04.532845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6693,"log_dups_size":0,"ondisk_log_size":6693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'9962","reported_seq":15919,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:04.538441+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:21:04.538441+0000","last_peered":"2023-10-14T01:21:04.538441+0000","last_clean":"2023-10-14T01:21:04.538441+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:21:04.538441+0000","last_undegraded":"2023-10-14T01:21:04.538441+0000","last_fullsized":"2023-10-14T01:21:04.538441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"126'6667","reported_seq":9655,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:04.534590+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:21:04.534590+0000","last_peered":"2023-10-14T01:21:04.534590+0000","last_clean":"2023-10-14T01:21:04.534590+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:21:04.534590+0000","last_undegraded":"2023-10-14T01:21:04.534590+0000","last_fullsized":"2023-10-14T01:21:04.534590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'16433","reported_seq":24362,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:04.549534+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:21:04.549534+0000","last_peered":"2023-10-14T01:21:04.549534+0000","last_clean":"2023-10-14T01:21:04.549534+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:21:04.549534+0000","last_undegraded":"2023-10-14T01:21:04.549534+0000","last_fullsized":"2023-10-14T01:21:04.549534+0000","mapping_epoch":12,"log_start":"89'6400","ondisk_log_start":"89'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'524815","reported_seq":1043369,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:00.903620+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:21:00.903620+0000","last_peered":"2023-10-14T01:21:00.903620+0000","last_clean":"2023-10-14T01:21:00.903620+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:21:00.903620+0000","last_undegraded":"2023-10-14T01:21:00.903620+0000","last_fullsized":"2023-10-14T01:21:00.903620+0000","mapping_epoch":15,"log_start":"125'514736","ondisk_log_start":"125'514736","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2908,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 555s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7140769792,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":520567,"num_read_kb":2050924,"num_write":520296,"num_write_kb":2081184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"126'3481","reported_seq":5160,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:57.373910+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:20:57.373910+0000","last_peered":"2023-10-14T01:20:57.373910+0000","last_clean":"2023-10-14T01:20:57.373910+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:20:57.373910+0000","last_undegraded":"2023-10-14T01:20:57.373910+0000","last_fullsized":"2023-10-14T01:20:57.373910+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'497068","reported_seq":990090,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:05.044347+0000","last_change":"2023-10-14T01:21:02.577185+0000","last_active":"2023-10-14T01:21:05.044347+0000","last_peered":"2023-10-14T01:21:05.044347+0000","last_clean":"2023-10-14T01:21:05.044347+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:21:05.044347+0000","last_undegraded":"2023-10-14T01:21:05.044347+0000","last_fullsized":"2023-10-14T01:21:05.044347+0000","mapping_epoch":15,"log_start":"125'487023","ondisk_log_start":"125'487023","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6784200704,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":495031,"num_read_kb":1949800,"num_write":494672,"num_write_kb":1978688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3032,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.939367+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:21:01.939367+0000","last_peered":"2023-10-14T01:21:01.939367+0000","last_clean":"2023-10-14T01:21:01.939367+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:21:01.939367+0000","last_undegraded":"2023-10-14T01:21:01.939367+0000","last_fullsized":"2023-10-14T01:21:01.939367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'518005","reported_seq":1029643,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:00.897656+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:21:00.897656+0000","last_peered":"2023-10-14T01:21:00.897656+0000","last_clean":"2023-10-14T01:21:00.897656+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:21:00.897656+0000","last_undegraded":"2023-10-14T01:21:00.897656+0000","last_fullsized":"2023-10-14T01:21:00.897656+0000","mapping_epoch":15,"log_start":"125'507928","ondisk_log_start":"125'507928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3137,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 556s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7032975360,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":513574,"num_read_kb":2023196,"num_write":513372,"num_write_kb":2053488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3182,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.950784+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:21:01.950784+0000","last_peered":"2023-10-14T01:21:01.950784+0000","last_clean":"2023-10-14T01:21:01.950784+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:21:01.950784+0000","last_undegraded":"2023-10-14T01:21:01.950784+0000","last_fullsized":"2023-10-14T01:21:01.950784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'492926","reported_seq":979815,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:05.052841+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:21:05.052841+0000","last_peered":"2023-10-14T01:21:05.052841+0000","last_clean":"2023-10-14T01:21:05.052841+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:21:05.052841+0000","last_undegraded":"2023-10-14T01:21:05.052841+0000","last_fullsized":"2023-10-14T01:21:05.052841+0000","mapping_epoch":15,"log_start":"125'482827","ondisk_log_start":"125'482827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3335,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 539s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6689099795,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":488701,"num_read_kb":1924384,"num_write":488279,"num_write_kb":1953110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"126'9909","reported_seq":14283,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:21:04.594615+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:21:04.594615+0000","last_peered":"2023-10-14T01:21:04.594615+0000","last_clean":"2023-10-14T01:21:04.594615+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:21:04.594615+0000","last_undegraded":"2023-10-14T01:21:04.594615+0000","last_fullsized":"2023-10-14T01:21:04.594615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":301,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T01:19:39.503510+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:19:39.503510+0000","last_peered":"2023-10-14T01:19:39.503510+0000","last_clean":"2023-10-14T01:19:39.503510+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:19:39.503510+0000","last_undegraded":"2023-10-14T01:19:39.503510+0000","last_fullsized":"2023-10-14T01:19:39.503510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27647045651,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2017873,"num_read_kb":7948304,"num_write":2016619,"num_write_kb":8066470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16522530816,"data_stored":25704894502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40300,"ondisk_log_size":40300,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":46969,"ondisk_log_size":46969,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739282,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9305160,"kb_used_data":8069248,"kb_used_omap":1270,"kb_used_meta":1234633,"kb_avail":85066680,"statfs":{"total":96636764160,"available":87108280320,"internally_reserved":0,"allocated":8262909952,"data_stored":12853262017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1264264836},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,3,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.372,"15min":0.42099999999999999},"min":{"1min":0.17399999999999999,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":0.42099999999999999,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.48299999999999998,"15min":0.54600000000000004},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.9820000000000002,"5min":7.9820000000000002,"15min":27.538},"last":0.26800000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.59799999999999998,"15min":0.51100000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":1.6299999999999999,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.65400000000000003,"15min":0.50600000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.6160000000000001,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.44400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6008560,"kb_used_data":3933480,"kb_used_omap":472,"kb_used_meta":2074599,"kb_avail":88363280,"statfs":{"total":96636764160,"available":90483998720,"internally_reserved":0,"allocated":4027883520,"data_stored":6268310341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":483633,"internal_metadata":2124390095},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.83799999999999997,"15min":0.59699999999999998},"min":{"1min":0.187,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":4.1070000000000002,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.218},{"interface":"front","average":{"1min":0.375,"5min":0.66600000000000004,"15min":0.59499999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":0.73399999999999999,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.61499999999999999,"15min":0.50900000000000001},"min":{"1min":0.20599999999999999,"5min":0.185,"15min":0.182},"max":{"1min":3.246,"5min":5.4790000000000001,"15min":15.891},"last":0.188},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.75700000000000001,"15min":0.59099999999999997},"min":{"1min":0.25700000000000001,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.76400000000000001,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.251}]}]},{"osd":0,"up_from":8,"seq":34359739280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5610992,"kb_used_data":4135976,"kb_used_omap":502,"kb_used_meta":1474505,"kb_avail":88760848,"statfs":{"total":96636764160,"available":90891108352,"internally_reserved":0,"allocated":4235239424,"data_stored":6584068834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1509894101},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,4,6,11,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.46300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20499999999999999,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":0.61199999999999999,"5min":4.867,"15min":5.7729999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":5.2830000000000004,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.438,"15min":0.48199999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":0.45400000000000001,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.42299999999999999,"15min":0.47699999999999998},"min":{"1min":0.24199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.64700000000000002,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4233908224,"data_stored":6583635968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4026175488,"data_stored":6267629587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8262447104,"data_stored":12853628947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:21:05.678 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:05.678 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:05.679 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:05.679 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:05.679 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:10.410 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.401+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:10.419 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:10.420 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:21:10.420 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:21:10.421 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:21:10.421 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:21:10.421 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:10.421 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.409+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:10.422 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.413+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:21:10.422 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:10.417+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:21:20.159 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.157+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:21:20.160 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.157+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:21:20.165 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:20.165 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:21:20.166 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:20.166 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:21:20.166 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:21:20.167 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:21:20.167 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:21:20.167 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:21:20.168 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:20.168 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:20.168 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:21:20.168 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:21:20.169 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:20.165+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:21:25.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:26.173 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:21:26.174 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:21:26.200 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2413,"stamp":"2023-10-14T01:21:25.524554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27676278501,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2059291,"num_read_kb":8007666,"num_write":2024780,"num_write_kb":8098179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87505,"ondisk_log_size":87505,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20594136,"kb_used_data":16198144,"kb_used_omap":2246,"kb_used_meta":4393721,"kb_avail":262521384,"statfs":{"total":289910292480,"available":268821897216,"internally_reserved":0,"allocated":16586899456,"data_stored":25766484067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":4499170969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,3,9,16,11,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14209024,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3829,"num_read_kb":15260,"num_write":3663,"num_write_kb":14652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001661"},"pg_stats":[{"pgid":"2.7","version":"127'6715","reported_seq":9912,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.416156+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:21:22.416156+0000","last_peered":"2023-10-14T01:21:22.416156+0000","last_clean":"2023-10-14T01:21:22.416156+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:21:22.416156+0000","last_undegraded":"2023-10-14T01:21:22.416156+0000","last_fullsized":"2023-10-14T01:21:22.416156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6715,"log_dups_size":0,"ondisk_log_size":6715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'9995","reported_seq":15966,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.415857+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:21:22.415857+0000","last_peered":"2023-10-14T01:21:22.415857+0000","last_clean":"2023-10-14T01:21:22.415857+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:21:22.415857+0000","last_undegraded":"2023-10-14T01:21:22.415857+0000","last_fullsized":"2023-10-14T01:21:22.415857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"127'6689","reported_seq":9686,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.415525+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:21:22.415525+0000","last_peered":"2023-10-14T01:21:22.415525+0000","last_clean":"2023-10-14T01:21:22.415525+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:21:22.415525+0000","last_undegraded":"2023-10-14T01:21:22.415525+0000","last_fullsized":"2023-10-14T01:21:22.415525+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'16488","reported_seq":24438,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.416546+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:21:22.416546+0000","last_peered":"2023-10-14T01:21:22.416546+0000","last_clean":"2023-10-14T01:21:22.416546+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:21:22.416546+0000","last_undegraded":"2023-10-14T01:21:22.416546+0000","last_fullsized":"2023-10-14T01:21:22.416546+0000","mapping_epoch":12,"log_start":"89'6400","ondisk_log_start":"89'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'526817","reported_seq":1047389,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:20.986480+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:21:20.986480+0000","last_peered":"2023-10-14T01:21:20.986480+0000","last_clean":"2023-10-14T01:21:20.986480+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:21:20.986480+0000","last_undegraded":"2023-10-14T01:21:20.986480+0000","last_fullsized":"2023-10-14T01:21:20.986480+0000","mapping_epoch":15,"log_start":"125'516736","ondisk_log_start":"125'516736","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3032,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 575s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7148298240,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":522571,"num_read_kb":2058900,"num_write":522211,"num_write_kb":2088844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"127'3493","reported_seq":5177,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:17.977172+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:21:17.977172+0000","last_peered":"2023-10-14T01:21:17.977172+0000","last_clean":"2023-10-14T01:21:17.977172+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:21:17.977172+0000","last_undegraded":"2023-10-14T01:21:17.977172+0000","last_fullsized":"2023-10-14T01:21:17.977172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3493,"log_dups_size":0,"ondisk_log_size":3493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'498886","reported_seq":993735,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:25.149528+0000","last_change":"2023-10-14T01:21:24.564643+0000","last_active":"2023-10-14T01:21:25.149528+0000","last_peered":"2023-10-14T01:21:25.149528+0000","last_clean":"2023-10-14T01:21:25.149528+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:21:25.149528+0000","last_undegraded":"2023-10-14T01:21:25.149528+0000","last_fullsized":"2023-10-14T01:21:25.149528+0000","mapping_epoch":15,"log_start":"125'488823","ondisk_log_start":"125'488823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6791299072,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":496849,"num_read_kb":1957028,"num_write":496490,"num_write_kb":1985960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3041,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.441916+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:21:22.441916+0000","last_peered":"2023-10-14T01:21:22.441916+0000","last_clean":"2023-10-14T01:21:22.441916+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:21:22.441916+0000","last_undegraded":"2023-10-14T01:21:22.441916+0000","last_fullsized":"2023-10-14T01:21:22.441916+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'519957","reported_seq":1033643,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:20.978241+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:21:20.978241+0000","last_peered":"2023-10-14T01:21:20.978241+0000","last_clean":"2023-10-14T01:21:20.978241+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:21:20.978241+0000","last_undegraded":"2023-10-14T01:21:20.978241+0000","last_fullsized":"2023-10-14T01:21:20.978241+0000","mapping_epoch":15,"log_start":"125'509928","ondisk_log_start":"125'509928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3289,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 576s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7040069632,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":515602,"num_read_kb":2031280,"num_write":515220,"num_write_kb":2060880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3191,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.415153+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:21:22.415153+0000","last_peered":"2023-10-14T01:21:22.415153+0000","last_clean":"2023-10-14T01:21:22.415153+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:21:22.415153+0000","last_undegraded":"2023-10-14T01:21:22.415153+0000","last_fullsized":"2023-10-14T01:21:22.415153+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'494814","reported_seq":983539,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:25.153851+0000","last_change":"2023-10-14T01:12:05.530412+0000","last_active":"2023-10-14T01:21:25.153851+0000","last_peered":"2023-10-14T01:21:25.153851+0000","last_clean":"2023-10-14T01:21:25.153851+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:21:25.153851+0000","last_undegraded":"2023-10-14T01:21:25.153851+0000","last_fullsized":"2023-10-14T01:21:25.153851+0000","mapping_epoch":15,"log_start":"125'484727","ondisk_log_start":"125'484727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3483,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 559s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6696017939,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":490521,"num_read_kb":1931652,"num_write":490063,"num_write_kb":1960246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"127'9942","reported_seq":14329,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:22.416470+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:21:22.416470+0000","last_peered":"2023-10-14T01:21:22.416470+0000","last_clean":"2023-10-14T01:21:22.416470+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:21:22.416470+0000","last_undegraded":"2023-10-14T01:21:22.416470+0000","last_fullsized":"2023-10-14T01:21:22.416470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":302,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-14T01:20:11.325495+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:20:11.325495+0000","last_peered":"2023-10-14T01:20:11.325495+0000","last_clean":"2023-10-14T01:20:11.325495+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:20:11.325495+0000","last_undegraded":"2023-10-14T01:20:11.325495+0000","last_fullsized":"2023-10-14T01:20:11.325495+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27675684883,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2025543,"num_read_kb":7978860,"num_write":2023984,"num_write_kb":8095930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16582750208,"data_stored":25765113894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40260,"ondisk_log_size":40260,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47146,"ondisk_log_size":47146,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739286,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9425192,"kb_used_data":8099040,"kb_used_omap":1270,"kb_used_meta":1324873,"kb_avail":84946648,"statfs":{"total":96636764160,"available":86985367552,"internally_reserved":0,"allocated":8293416960,"data_stored":12883761130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1356670596},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.372,"15min":0.42099999999999999},"min":{"1min":0.17399999999999999,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":0.42099999999999999,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.48299999999999998,"15min":0.54600000000000004},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.9820000000000002,"5min":7.9820000000000002,"15min":27.538},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.59799999999999998,"15min":0.51100000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":1.6299999999999999,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.65400000000000003,"15min":0.50600000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.6160000000000001,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5481796,"kb_used_data":3947836,"kb_used_omap":473,"kb_used_meta":1533478,"kb_avail":88890044,"statfs":{"total":96636764160,"available":91023405056,"internally_reserved":0,"allocated":4042584064,"data_stored":6283002990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1570281536},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.83799999999999997,"15min":0.59699999999999998},"min":{"1min":0.187,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":4.1070000000000002,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.183},{"interface":"front","average":{"1min":0.375,"5min":0.66600000000000004,"15min":0.59499999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":0.73399999999999999,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.61499999999999999,"15min":0.50900000000000001},"min":{"1min":0.20599999999999999,"5min":0.185,"15min":0.182},"max":{"1min":3.246,"5min":5.4790000000000001,"15min":15.891},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.75700000000000001,"15min":0.59099999999999997},"min":{"1min":0.25700000000000001,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.76400000000000001,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.58799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739284,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5687148,"kb_used_data":4151268,"kb_used_omap":502,"kb_used_meta":1535369,"kb_avail":88684692,"statfs":{"total":96636764160,"available":90813124608,"internally_reserved":0,"allocated":4250898432,"data_stored":6599719947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1572218837},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,11,11,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.46300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20499999999999999,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":0.61199999999999999,"5min":4.867,"15min":5.7729999999999997},"last":0.376},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":5.2830000000000004,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.438,"15min":0.48199999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":0.45400000000000001,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.42299999999999999,"15min":0.47699999999999998},"min":{"1min":0.24199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.64700000000000002,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4249317376,"data_stored":6599045120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4040916992,"data_stored":6282371091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8292515840,"data_stored":12883697683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:21:26.201 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:26.202 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:26.202 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:26.203 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:26.203 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:29.609 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.601+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:29.610 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.609+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:29.610 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.609+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:21:29.611 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.609+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:21:29.633 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.625+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:21:29.633 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.625+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:21:29.634 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.625+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:29.634 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.625+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:29.635 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.625+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:21:29.635 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.629+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:21:29.636 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.629+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:29.636 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.629+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:29.636 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.629+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:29.637 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:29.629+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:34.104 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.097+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:21:34.105 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.097+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:21:34.112 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.109+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:34.113 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.109+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:34.113 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.109+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:21:34.113 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.109+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:21:34.116 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.113+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:21:34.116 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.113+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:21:34.116 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.113+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:34.117 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.113+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:34.119 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:34.117+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:21:38.187 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.185+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:21:38.187 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.185+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:21:38.187 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.185+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:21:38.188 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.185+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:21:38.197 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.189+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:21:38.197 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.189+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:21:38.197 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.189+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:21:38.217 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:21:38.209+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:21:46.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:46.729 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:21:46.730 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:21:46.771 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2423,"stamp":"2023-10-14T01:21:45.527973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27705478885,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2066963,"num_read_kb":8038198,"num_write":2032273,"num_write_kb":8128151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87604,"ondisk_log_size":87604,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20827848,"kb_used_data":16257520,"kb_used_omap":2246,"kb_used_meta":4568057,"kb_avail":262287672,"statfs":{"total":289910292480,"available":268582576128,"internally_reserved":0,"allocated":16647700480,"data_stored":25827285091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":4677691033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,3,11,10,15,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18325504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4888,"num_read_kb":19456,"num_write":4708,"num_write_kb":18832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001997"},"pg_stats":[{"pgid":"2.7","version":"127'6758","reported_seq":9964,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:45.062978+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:21:45.062978+0000","last_peered":"2023-10-14T01:21:45.062978+0000","last_clean":"2023-10-14T01:21:45.062978+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:21:45.062978+0000","last_undegraded":"2023-10-14T01:21:45.062978+0000","last_fullsized":"2023-10-14T01:21:45.062978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'10059","reported_seq":16042,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:45.079194+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:21:45.079194+0000","last_peered":"2023-10-14T01:21:45.079194+0000","last_clean":"2023-10-14T01:21:45.079194+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:21:45.079194+0000","last_undegraded":"2023-10-14T01:21:45.079194+0000","last_fullsized":"2023-10-14T01:21:45.079194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"127'6732","reported_seq":9737,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:45.064386+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:21:45.064386+0000","last_peered":"2023-10-14T01:21:45.064386+0000","last_clean":"2023-10-14T01:21:45.064386+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:21:45.064386+0000","last_undegraded":"2023-10-14T01:21:45.064386+0000","last_fullsized":"2023-10-14T01:21:45.064386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'16590","reported_seq":24560,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:45.099599+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:21:45.099599+0000","last_peered":"2023-10-14T01:21:45.099599+0000","last_clean":"2023-10-14T01:21:45.099599+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:21:45.099599+0000","last_undegraded":"2023-10-14T01:21:45.099599+0000","last_fullsized":"2023-10-14T01:21:45.099599+0000","mapping_epoch":12,"log_start":"90'6500","ondisk_log_start":"90'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'528791","reported_seq":1051371,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:41.120904+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:21:41.120904+0000","last_peered":"2023-10-14T01:21:41.120904+0000","last_clean":"2023-10-14T01:21:41.120904+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:21:41.120904+0000","last_undegraded":"2023-10-14T01:21:41.120904+0000","last_fullsized":"2023-10-14T01:21:41.120904+0000","mapping_epoch":15,"log_start":"125'518736","ondisk_log_start":"125'518736","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3181,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 595s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7155576832,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":524568,"num_read_kb":2066864,"num_write":524084,"num_write_kb":2096336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"127'3504","reported_seq":5192,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:39.604557+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:21:39.604557+0000","last_peered":"2023-10-14T01:21:39.604557+0000","last_clean":"2023-10-14T01:21:39.604557+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:21:39.604557+0000","last_undegraded":"2023-10-14T01:21:39.604557+0000","last_fullsized":"2023-10-14T01:21:39.604557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3504,"log_dups_size":0,"ondisk_log_size":3504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'500725","reported_seq":997437,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:45.283611+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:21:45.283611+0000","last_peered":"2023-10-14T01:21:45.283611+0000","last_clean":"2023-10-14T01:21:45.283611+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:21:45.283611+0000","last_undegraded":"2023-10-14T01:21:45.283611+0000","last_fullsized":"2023-10-14T01:21:45.283611+0000","mapping_epoch":15,"log_start":"125'490723","ondisk_log_start":"125'490723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":38,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6798426112,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":498701,"num_read_kb":1964404,"num_write":498304,"num_write_kb":1993216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3049,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:42.909766+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:21:42.909766+0000","last_peered":"2023-10-14T01:21:42.909766+0000","last_clean":"2023-10-14T01:21:42.909766+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:21:42.909766+0000","last_undegraded":"2023-10-14T01:21:42.909766+0000","last_fullsized":"2023-10-14T01:21:42.909766+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'521975","reported_seq":1037602,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:41.115894+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:21:41.115894+0000","last_peered":"2023-10-14T01:21:41.115894+0000","last_clean":"2023-10-14T01:21:41.115894+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:21:41.115894+0000","last_undegraded":"2023-10-14T01:21:41.115894+0000","last_fullsized":"2023-10-14T01:21:41.115894+0000","mapping_epoch":15,"log_start":"125'511928","ondisk_log_start":"125'511928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3409,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 596s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7047639040,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":517534,"num_read_kb":2038952,"num_write":517156,"num_write_kb":2068624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3200,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:42.896184+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:21:42.896184+0000","last_peered":"2023-10-14T01:21:42.896184+0000","last_clean":"2023-10-14T01:21:42.896184+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:21:42.896184+0000","last_undegraded":"2023-10-14T01:21:42.896184+0000","last_fullsized":"2023-10-14T01:21:42.896184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'496757","reported_seq":987385,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:45.307507+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:21:45.307507+0000","last_peered":"2023-10-14T01:21:45.307507+0000","last_clean":"2023-10-14T01:21:45.307507+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:21:45.307507+0000","last_undegraded":"2023-10-14T01:21:45.307507+0000","last_fullsized":"2023-10-14T01:21:45.307507+0000","mapping_epoch":15,"log_start":"125'486727","ondisk_log_start":"125'486727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6703243283,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":492412,"num_read_kb":1939172,"num_write":491933,"num_write_kb":1967726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"127'10004","reported_seq":14403,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:45.124208+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:21:45.124208+0000","last_peered":"2023-10-14T01:21:45.124208+0000","last_clean":"2023-10-14T01:21:45.124208+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:21:45.124208+0000","last_undegraded":"2023-10-14T01:21:45.124208+0000","last_fullsized":"2023-10-14T01:21:45.124208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":303,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:16.344456+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:21:16.344456+0000","last_peered":"2023-10-14T01:21:16.344456+0000","last_clean":"2023-10-14T01:21:16.344456+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:21:16.344456+0000","last_undegraded":"2023-10-14T01:21:16.344456+0000","last_fullsized":"2023-10-14T01:21:16.344456+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27704885267,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2033215,"num_read_kb":8009392,"num_write":2031477,"num_write_kb":8125902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16644272128,"data_stored":25826635814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40134,"ondisk_log_size":40134,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47371,"ondisk_log_size":47371,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739290,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9550536,"kb_used_data":8128512,"kb_used_omap":1270,"kb_used_meta":1420745,"kb_avail":84821304,"statfs":{"total":96636764160,"available":86857015296,"internally_reserved":0,"allocated":8323596288,"data_stored":12913940458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1454843524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:41 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.372,"15min":0.42099999999999999},"min":{"1min":0.17399999999999999,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":0.42099999999999999,"5min":4.2560000000000002,"15min":27.402000000000001},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.48299999999999998,"15min":0.54600000000000004},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.9820000000000002,"5min":7.9820000000000002,"15min":27.538},"last":0.27200000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.59799999999999998,"15min":0.51100000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":1.6299999999999999,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.65400000000000003,"15min":0.50600000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.6160000000000001,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5556996,"kb_used_data":3962620,"kb_used_omap":473,"kb_used_meta":1593894,"kb_avail":88814844,"statfs":{"total":96636764160,"available":90946400256,"internally_reserved":0,"allocated":4057722880,"data_stored":6298141806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1632147520},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,3,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.83799999999999997,"15min":0.59699999999999998},"min":{"1min":0.187,"5min":0.184,"15min":0.17100000000000001},"max":{"1min":4.1070000000000002,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.66600000000000004,"15min":0.59499999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":0.73399999999999999,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.61499999999999999,"15min":0.50900000000000001},"min":{"1min":0.20599999999999999,"5min":0.185,"15min":0.182},"max":{"1min":3.246,"5min":5.4790000000000001,"15min":15.891},"last":0.247},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.75700000000000001,"15min":0.59099999999999997},"min":{"1min":0.25700000000000001,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.76400000000000001,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5720316,"kb_used_data":4166388,"kb_used_omap":502,"kb_used_meta":1553417,"kb_avail":88651524,"statfs":{"total":96636764160,"available":90779160576,"internally_reserved":0,"allocated":4266381312,"data_stored":6615202827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1590699989},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,2,12,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.46300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20499999999999999,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":0.61199999999999999,"5min":4.867,"15min":5.7729999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.214,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":5.2830000000000004,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:20:50 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.438,"15min":0.48199999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":0.45400000000000001,"5min":3.1539999999999999,"15min":5.1909999999999998},"last":4.1130000000000004},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.42299999999999999,"15min":0.47699999999999998},"min":{"1min":0.24199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.64700000000000002,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4264923136,"data_stored":6614650880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4056027136,"data_stored":6297481235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8323321856,"data_stored":12914503699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:21:46.773 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:46.774 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:46.774 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:46.774 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:06.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:07.297 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:22:07.298 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:22:07.345 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2433,"stamp":"2023-10-14T01:22:05.533283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27734875877,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2074459,"num_read_kb":8068006,"num_write":2039833,"num_write_kb":8158391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87869,"ondisk_log_size":87869,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21081240,"kb_used_data":16318016,"kb_used_omap":2246,"kb_used_meta":4760953,"kb_avail":262034280,"statfs":{"total":289910292480,"available":268323102720,"internally_reserved":0,"allocated":16709648384,"data_stored":25889232995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":4875216537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,4,9,7,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18190336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4667,"num_read_kb":18548,"num_write":4679,"num_write_kb":18716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002027"},"pg_stats":[{"pgid":"2.7","version":"127'6796","reported_seq":10010,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:04.301306+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:22:04.301306+0000","last_peered":"2023-10-14T01:22:04.301306+0000","last_clean":"2023-10-14T01:22:04.301306+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:22:04.301306+0000","last_undegraded":"2023-10-14T01:22:04.301306+0000","last_fullsized":"2023-10-14T01:22:04.301306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6796,"log_dups_size":0,"ondisk_log_size":6796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'10116","reported_seq":16111,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:04.313182+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:22:04.313182+0000","last_peered":"2023-10-14T01:22:04.313182+0000","last_clean":"2023-10-14T01:22:04.313182+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:22:04.313182+0000","last_undegraded":"2023-10-14T01:22:04.313182+0000","last_fullsized":"2023-10-14T01:22:04.313182+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"127'6770","reported_seq":9783,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:04.301389+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:22:04.301389+0000","last_peered":"2023-10-14T01:22:04.301389+0000","last_clean":"2023-10-14T01:22:04.301389+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:22:04.301389+0000","last_undegraded":"2023-10-14T01:22:04.301389+0000","last_fullsized":"2023-10-14T01:22:04.301389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'16685","reported_seq":24675,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:04.328858+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:22:04.328858+0000","last_peered":"2023-10-14T01:22:04.328858+0000","last_clean":"2023-10-14T01:22:04.328858+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:22:04.328858+0000","last_undegraded":"2023-10-14T01:22:04.328858+0000","last_fullsized":"2023-10-14T01:22:04.328858+0000","mapping_epoch":12,"log_start":"94'6600","ondisk_log_start":"94'6600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'530734","reported_seq":1055203,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:01.205854+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:22:01.205854+0000","last_peered":"2023-10-14T01:22:01.205854+0000","last_clean":"2023-10-14T01:22:01.205854+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:22:01.205854+0000","last_undegraded":"2023-10-14T01:22:01.205854+0000","last_fullsized":"2023-10-14T01:22:01.205854+0000","mapping_epoch":15,"log_start":"126'520636","ondisk_log_start":"126'520636","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3272,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 616s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7162974208,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":526447,"num_read_kb":2074328,"num_write":525965,"num_write_kb":2103860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"127'3512","reported_seq":5204,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:58.260664+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:21:58.260664+0000","last_peered":"2023-10-14T01:21:58.260664+0000","last_clean":"2023-10-14T01:21:58.260664+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:21:58.260664+0000","last_undegraded":"2023-10-14T01:21:58.260664+0000","last_fullsized":"2023-10-14T01:21:58.260664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3512,"log_dups_size":0,"ondisk_log_size":3512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'502677","reported_seq":1001281,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:05.394257+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:22:05.394257+0000","last_peered":"2023-10-14T01:22:05.394257+0000","last_clean":"2023-10-14T01:22:05.394257+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:22:05.394257+0000","last_undegraded":"2023-10-14T01:22:05.394257+0000","last_fullsized":"2023-10-14T01:22:05.394257+0000","mapping_epoch":15,"log_start":"126'492623","ondisk_log_start":"126'492623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":153,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 25s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6805696512,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":500587,"num_read_kb":1971908,"num_write":500176,"num_write_kb":2000704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3057,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:03.383477+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:22:03.383477+0000","last_peered":"2023-10-14T01:22:03.383477+0000","last_clean":"2023-10-14T01:22:03.383477+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:22:03.383477+0000","last_undegraded":"2023-10-14T01:22:03.383477+0000","last_fullsized":"2023-10-14T01:22:03.383477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'524012","reported_seq":1041529,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:01.210822+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:22:01.210822+0000","last_peered":"2023-10-14T01:22:01.210822+0000","last_clean":"2023-10-14T01:22:01.210822+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:22:01.210822+0000","last_undegraded":"2023-10-14T01:22:01.210822+0000","last_fullsized":"2023-10-14T01:22:01.210822+0000","mapping_epoch":15,"log_start":"126'513928","ondisk_log_start":"126'513928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3513,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 617s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7055233024,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":519413,"num_read_kb":2046436,"num_write":519123,"num_write_kb":2076492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3208,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:03.376784+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:22:03.376784+0000","last_peered":"2023-10-14T01:22:03.376784+0000","last_clean":"2023-10-14T01:22:03.376784+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:22:03.376784+0000","last_undegraded":"2023-10-14T01:22:03.376784+0000","last_fullsized":"2023-10-14T01:22:03.376784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'498597","reported_seq":991078,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:05.404817+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:22:05.404817+0000","last_peered":"2023-10-14T01:22:05.404817+0000","last_clean":"2023-10-14T01:22:05.404817+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:22:05.404817+0000","last_undegraded":"2023-10-14T01:22:05.404817+0000","last_fullsized":"2023-10-14T01:22:05.404817+0000","mapping_epoch":15,"log_start":"126'488527","ondisk_log_start":"126'488527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6710378515,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":494264,"num_read_kb":1946528,"num_write":493773,"num_write_kb":1975086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"127'10061","reported_seq":14472,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:04.340241+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:22:04.340241+0000","last_peered":"2023-10-14T01:22:04.340241+0000","last_clean":"2023-10-14T01:22:04.340241+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:22:04.340241+0000","last_undegraded":"2023-10-14T01:22:04.340241+0000","last_fullsized":"2023-10-14T01:22:04.340241+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":303,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:16.344456+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:21:16.344456+0000","last_peered":"2023-10-14T01:21:16.344456+0000","last_clean":"2023-10-14T01:21:16.344456+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:21:16.344456+0000","last_undegraded":"2023-10-14T01:21:16.344456+0000","last_fullsized":"2023-10-14T01:21:16.344456+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27734282259,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2040711,"num_read_kb":8039200,"num_write":2039037,"num_write_kb":8156142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16706187264,"data_stored":25888550950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40306,"ondisk_log_size":40306,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47464,"ondisk_log_size":47464,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739294,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9676688,"kb_used_data":8158664,"kb_used_omap":1270,"kb_used_meta":1516745,"kb_avail":84695152,"statfs":{"total":96636764160,"available":86727835648,"internally_reserved":0,"allocated":8354471936,"data_stored":12944816106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1553147524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.32600000000000001,"15min":0.41699999999999998},"min":{"1min":0.184,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":0.36499999999999999,"5min":2.5840000000000001,"15min":27.402000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.42199999999999999,"15min":0.53000000000000003},"min":{"1min":0.20999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41799999999999998,"5min":7.9820000000000002,"15min":27.538},"last":0.23999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.57799999999999996,"15min":0.48999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.53900000000000003,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.33900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5608076,"kb_used_data":3977668,"kb_used_omap":473,"kb_used_meta":1629926,"kb_avail":88763764,"statfs":{"total":96636764160,"available":90894094336,"internally_reserved":0,"allocated":4073132032,"data_stored":6313550958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1669044288},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:55 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.81499999999999995,"15min":0.59399999999999997},"min":{"1min":0.183,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":0.52500000000000002,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.67700000000000005,"15min":0.59899999999999998},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":2.012,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.315}]},{"osd":2,"last update":"Sat Oct 14 01:21:55 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.52800000000000002,"15min":0.501},"min":{"1min":0.188,"5min":0.185,"15min":0.184},"max":{"1min":0.81799999999999995,"5min":5.4790000000000001,"15min":15.891},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.748,"15min":0.58799999999999997},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.58799999999999997,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.48499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5796476,"kb_used_data":4181684,"kb_used_omap":502,"kb_used_meta":1614281,"kb_avail":88575364,"statfs":{"total":96636764160,"available":90701172736,"internally_reserved":0,"allocated":4282044416,"data_stored":6630865931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1653024725},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":0.50700000000000001,"5min":4.867,"15min":5.7729999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.48299999999999998,"15min":0.49099999999999999},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":1.1970000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.442,"15min":0.48599999999999999},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.41799999999999998,"15min":0.47499999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61099999999999999,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4280700928,"data_stored":6630428672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4071211008,"data_stored":6312665107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8354275328,"data_stored":12945457171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:22:07.347 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:07.348 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:07.348 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:07.349 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:27.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:27.883 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:22:27.884 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:22:27.916 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2445,"stamp":"2023-10-14T01:22:27.544515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27768278757,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2082999,"num_read_kb":8101958,"num_write":2048424,"num_write_kb":8192755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87923,"ondisk_log_size":87923,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21326124,"kb_used_data":16382484,"kb_used_omap":2246,"kb_used_meta":4941369,"kb_avail":261789396,"statfs":{"total":289910292480,"available":268072341504,"internally_reserved":0,"allocated":16775663616,"data_stored":25955248227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":5059962521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,8,8,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14815232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3782,"num_read_kb":15040,"num_write":3837,"num_write_kb":15348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002652"},"pg_stats":[{"pgid":"2.7","version":"127'6818","reported_seq":10040,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.859335+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:22:23.859335+0000","last_peered":"2023-10-14T01:22:23.859335+0000","last_clean":"2023-10-14T01:22:23.859335+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:22:23.859335+0000","last_undegraded":"2023-10-14T01:22:23.859335+0000","last_fullsized":"2023-10-14T01:22:23.859335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'10149","reported_seq":16156,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.859172+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:22:23.859172+0000","last_peered":"2023-10-14T01:22:23.859172+0000","last_clean":"2023-10-14T01:22:23.859172+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:22:23.859172+0000","last_undegraded":"2023-10-14T01:22:23.859172+0000","last_fullsized":"2023-10-14T01:22:23.859172+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"127'6792","reported_seq":9813,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.858955+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:22:23.858955+0000","last_peered":"2023-10-14T01:22:23.858955+0000","last_clean":"2023-10-14T01:22:23.858955+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:22:23.858955+0000","last_undegraded":"2023-10-14T01:22:23.858955+0000","last_fullsized":"2023-10-14T01:22:23.858955+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'16740","reported_seq":24750,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.860508+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:22:23.860508+0000","last_peered":"2023-10-14T01:22:23.860508+0000","last_clean":"2023-10-14T01:22:23.860508+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:22:23.860508+0000","last_undegraded":"2023-10-14T01:22:23.860508+0000","last_fullsized":"2023-10-14T01:22:23.860508+0000","mapping_epoch":12,"log_start":"94'6700","ondisk_log_start":"94'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'533335","reported_seq":1060345,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:26.315411+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:22:26.315411+0000","last_peered":"2023-10-14T01:22:26.315411+0000","last_clean":"2023-10-14T01:22:26.315411+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:22:26.315411+0000","last_undegraded":"2023-10-14T01:22:26.315411+0000","last_fullsized":"2023-10-14T01:22:26.315411+0000","mapping_epoch":15,"log_start":"126'523236","ondisk_log_start":"126'523236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3429,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 641s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7172726784,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":528975,"num_read_kb":2084376,"num_write":528458,"num_write_kb":2113832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"127'3529","reported_seq":5226,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:24.463971+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:22:24.463971+0000","last_peered":"2023-10-14T01:22:24.463971+0000","last_clean":"2023-10-14T01:22:24.463971+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:22:24.463971+0000","last_undegraded":"2023-10-14T01:22:24.463971+0000","last_fullsized":"2023-10-14T01:22:24.463971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'504577","reported_seq":1005003,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:25.492700+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:22:25.492700+0000","last_peered":"2023-10-14T01:22:25.492700+0000","last_clean":"2023-10-14T01:22:25.492700+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:22:25.492700+0000","last_undegraded":"2023-10-14T01:22:25.492700+0000","last_fullsized":"2023-10-14T01:22:25.492700+0000","mapping_epoch":15,"log_start":"126'494523","ondisk_log_start":"126'494523","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":272,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 45s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6812819456,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":502392,"num_read_kb":1979080,"num_write":501995,"num_write_kb":2007980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3066,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.860601+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:22:23.860601+0000","last_peered":"2023-10-14T01:22:23.860601+0000","last_clean":"2023-10-14T01:22:23.860601+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:22:23.860601+0000","last_undegraded":"2023-10-14T01:22:23.860601+0000","last_fullsized":"2023-10-14T01:22:23.860601+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'526480","reported_seq":1046380,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:26.336912+0000","last_change":"2023-10-14T01:11:44.130559+0000","last_active":"2023-10-14T01:22:26.336912+0000","last_peered":"2023-10-14T01:22:26.336912+0000","last_clean":"2023-10-14T01:22:26.336912+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:22:26.336912+0000","last_undegraded":"2023-10-14T01:22:26.336912+0000","last_fullsized":"2023-10-14T01:22:26.336912+0000","mapping_epoch":15,"log_start":"126'516428","ondisk_log_start":"126'516428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3648,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 642s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7064391680,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":521778,"num_read_kb":2055832,"num_write":521499,"num_write_kb":2085996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3216,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.859419+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:22:23.859419+0000","last_peered":"2023-10-14T01:22:23.859419+0000","last_clean":"2023-10-14T01:22:23.859419+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:22:23.859419+0000","last_undegraded":"2023-10-14T01:22:23.859419+0000","last_fullsized":"2023-10-14T01:22:23.859419+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'500500","reported_seq":994823,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:25.495203+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:22:25.495203+0000","last_peered":"2023-10-14T01:22:25.495203+0000","last_clean":"2023-10-14T01:22:25.495203+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:22:25.495203+0000","last_undegraded":"2023-10-14T01:22:25.495203+0000","last_fullsized":"2023-10-14T01:22:25.495203+0000","mapping_epoch":15,"log_start":"126'490427","ondisk_log_start":"126'490427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6717747219,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":496106,"num_read_kb":1953864,"num_write":495676,"num_write_kb":1982698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"127'10094","reported_seq":14517,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:22:23.859543+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:22:23.859543+0000","last_peered":"2023-10-14T01:22:23.859543+0000","last_clean":"2023-10-14T01:22:23.859543+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:22:23.859543+0000","last_undegraded":"2023-10-14T01:22:23.859543+0000","last_fullsized":"2023-10-14T01:22:23.859543+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":303,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:16.344456+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:21:16.344456+0000","last_peered":"2023-10-14T01:21:16.344456+0000","last_clean":"2023-10-14T01:21:16.344456+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:21:16.344456+0000","last_undegraded":"2023-10-14T01:21:16.344456+0000","last_fullsized":"2023-10-14T01:21:16.344456+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27767685139,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2049251,"num_read_kb":8073152,"num_write":2047628,"num_write_kb":8190506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16772415488,"data_stored":25954779174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40278,"ondisk_log_size":40278,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47546,"ondisk_log_size":47546,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9792940,"kb_used_data":8188900,"kb_used_omap":1270,"kb_used_meta":1602761,"kb_avail":84578900,"statfs":{"total":96636764160,"available":86608793600,"internally_reserved":0,"allocated":8385433600,"data_stored":12975777770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1641227908},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.32600000000000001,"15min":0.41699999999999998},"min":{"1min":0.184,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":0.36499999999999999,"5min":2.5840000000000001,"15min":27.402000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.42199999999999999,"15min":0.53000000000000003},"min":{"1min":0.20999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41799999999999998,"5min":7.9820000000000002,"15min":27.538},"last":0.254}]},{"osd":1,"last update":"Sat Oct 14 01:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.57799999999999996,"15min":0.48999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.53900000000000003,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.53200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739297,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5676912,"kb_used_data":3992360,"kb_used_omap":473,"kb_used_meta":1684070,"kb_avail":88694928,"statfs":{"total":96636764160,"available":90823606272,"internally_reserved":0,"allocated":4088176640,"data_stored":6328595566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1724487744},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:55 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.81499999999999995,"15min":0.59399999999999997},"min":{"1min":0.183,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":0.52500000000000002,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.217},{"interface":"front","average":{"1min":0.443,"5min":0.67700000000000005,"15min":0.59899999999999998},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":2.012,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.192}]},{"osd":2,"last update":"Sat Oct 14 01:21:55 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.52800000000000002,"15min":0.501},"min":{"1min":0.188,"5min":0.185,"15min":0.184},"max":{"1min":0.81799999999999995,"5min":5.4790000000000001,"15min":15.891},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.748,"15min":0.58799999999999997},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.58799999999999997,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856272,"kb_used_data":4201224,"kb_used_omap":502,"kb_used_meta":1654537,"kb_avail":88515568,"statfs":{"total":96636764160,"available":90639941632,"internally_reserved":0,"allocated":4302053376,"data_stored":6650874891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1694246869},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":0.50700000000000001,"5min":4.867,"15min":5.7729999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.48299999999999998,"15min":0.49099999999999999},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":1.1970000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.442,"15min":0.48599999999999999},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.41799999999999998,"15min":0.47499999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61099999999999999,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4300623872,"data_stored":6650351616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4086468608,"data_stored":6327922707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8385323008,"data_stored":12976504851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:22:27.918 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:27.919 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:27.919 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:27.920 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:36.322 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:22:36.316+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:22:36.329 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:22:36.324+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:22:36.344 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:22:36.336+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:22:36.345 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:22:36.336+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:22:36.345 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:22:36.336+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:22:36.346 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:22:36.344+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:22:47.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:48.497 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:22:48.498 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:22:48.525 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2455,"stamp":"2023-10-14T01:22:47.547862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27797917413,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2090596,"num_read_kb":8132170,"num_write":2056053,"num_write_kb":8223271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87847,"ondisk_log_size":87847,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21591824,"kb_used_data":16444472,"kb_used_omap":2246,"kb_used_meta":5145081,"kb_avail":261523696,"statfs":{"total":289910292480,"available":267800264704,"internally_reserved":0,"allocated":16839139328,"data_stored":26018712413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":5268563609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,11,11,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18710528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4789,"num_read_kb":19052,"num_write":4815,"num_write_kb":19260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001972"},"pg_stats":[{"pgid":"2.7","version":"128'6837","reported_seq":10068,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:45.064607+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:22:45.064607+0000","last_peered":"2023-10-14T01:22:45.064607+0000","last_clean":"2023-10-14T01:22:45.064607+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:22:45.064607+0000","last_undegraded":"2023-10-14T01:22:45.064607+0000","last_fullsized":"2023-10-14T01:22:45.064607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6837,"log_dups_size":0,"ondisk_log_size":6837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'10178","reported_seq":16198,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:45.067567+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:22:45.067567+0000","last_peered":"2023-10-14T01:22:45.067567+0000","last_clean":"2023-10-14T01:22:45.067567+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:22:45.067567+0000","last_undegraded":"2023-10-14T01:22:45.067567+0000","last_fullsized":"2023-10-14T01:22:45.067567+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"128'6812","reported_seq":9842,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:45.062586+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:22:45.062586+0000","last_peered":"2023-10-14T01:22:45.062586+0000","last_clean":"2023-10-14T01:22:45.062586+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:22:45.062586+0000","last_undegraded":"2023-10-14T01:22:45.062586+0000","last_fullsized":"2023-10-14T01:22:45.062586+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'16787","reported_seq":24818,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:45.082861+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:22:45.082861+0000","last_peered":"2023-10-14T01:22:45.082861+0000","last_clean":"2023-10-14T01:22:45.082861+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:22:45.082861+0000","last_undegraded":"2023-10-14T01:22:45.082861+0000","last_fullsized":"2023-10-14T01:22:45.082861+0000","mapping_epoch":12,"log_start":"94'6700","ondisk_log_start":"94'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'535371","reported_seq":1064317,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:46.463553+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:22:46.463553+0000","last_peered":"2023-10-14T01:22:46.463553+0000","last_clean":"2023-10-14T01:22:46.463553+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:22:46.463553+0000","last_undegraded":"2023-10-14T01:22:46.463553+0000","last_fullsized":"2023-10-14T01:22:46.463553+0000","mapping_epoch":15,"log_start":"126'525336","ondisk_log_start":"126'525336","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3564,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 661s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7180300288,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":530892,"num_read_kb":2092000,"num_write":530401,"num_write_kb":2121604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"128'3544","reported_seq":5246,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:44.334916+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:22:44.334916+0000","last_peered":"2023-10-14T01:22:44.334916+0000","last_clean":"2023-10-14T01:22:44.334916+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:22:44.334916+0000","last_undegraded":"2023-10-14T01:22:44.334916+0000","last_fullsized":"2023-10-14T01:22:44.334916+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'506557","reported_seq":1008885,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:45.574319+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:22:45.574319+0000","last_peered":"2023-10-14T01:22:45.574319+0000","last_clean":"2023-10-14T01:22:45.574319+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:22:45.574319+0000","last_undegraded":"2023-10-14T01:22:45.574319+0000","last_fullsized":"2023-10-14T01:22:45.574319+0000","mapping_epoch":15,"log_start":"126'496523","ondisk_log_start":"126'496523","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":363,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 65s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6820319232,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":504281,"num_read_kb":1986600,"num_write":503913,"num_write_kb":2015652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3075,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:44.336168+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:22:44.336168+0000","last_peered":"2023-10-14T01:22:44.336168+0000","last_clean":"2023-10-14T01:22:44.336168+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:22:44.336168+0000","last_undegraded":"2023-10-14T01:22:44.336168+0000","last_fullsized":"2023-10-14T01:22:44.336168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'528466","reported_seq":1050316,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:46.459302+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:22:46.459302+0000","last_peered":"2023-10-14T01:22:46.459302+0000","last_clean":"2023-10-14T01:22:46.459302+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:22:46.459302+0000","last_undegraded":"2023-10-14T01:22:46.459302+0000","last_fullsized":"2023-10-14T01:22:46.459302+0000","mapping_epoch":15,"log_start":"126'518428","ondisk_log_start":"126'518428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7071678464,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":523711,"num_read_kb":2063504,"num_write":523404,"num_write_kb":2093616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3225,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:44.335389+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:22:44.335389+0000","last_peered":"2023-10-14T01:22:44.335389+0000","last_clean":"2023-10-14T01:22:44.335389+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:22:44.335389+0000","last_undegraded":"2023-10-14T01:22:44.335389+0000","last_fullsized":"2023-10-14T01:22:44.335389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'502363","reported_seq":998546,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:45.584136+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:22:45.584136+0000","last_peered":"2023-10-14T01:22:45.584136+0000","last_clean":"2023-10-14T01:22:45.584136+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:22:45.584136+0000","last_undegraded":"2023-10-14T01:22:45.584136+0000","last_fullsized":"2023-10-14T01:22:45.584136+0000","mapping_epoch":15,"log_start":"126'492327","ondisk_log_start":"126'492327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6725025811,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":497964,"num_read_kb":1961260,"num_write":497539,"num_write_kb":1990150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"128'10123","reported_seq":14559,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:22:45.105048+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:22:45.105048+0000","last_peered":"2023-10-14T01:22:45.105048+0000","last_clean":"2023-10-14T01:22:45.105048+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:22:45.105048+0000","last_undegraded":"2023-10-14T01:22:45.105048+0000","last_fullsized":"2023-10-14T01:22:45.105048+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":304,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:40.263518+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:21:40.263518+0000","last_peered":"2023-10-14T01:21:40.263518+0000","last_clean":"2023-10-14T01:21:40.263518+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:21:40.263518+0000","last_undegraded":"2023-10-14T01:21:40.263518+0000","last_fullsized":"2023-10-14T01:21:40.263518+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27797323795,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2056848,"num_read_kb":8103364,"num_write":2055257,"num_write_kb":8221022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16834809856,"data_stored":26017173542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40143,"ondisk_log_size":40143,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47605,"ondisk_log_size":47605,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739302,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9932724,"kb_used_data":8220460,"kb_used_omap":1270,"kb_used_meta":1710985,"kb_avail":84439116,"statfs":{"total":96636764160,"available":86465654784,"internally_reserved":0,"allocated":8417751040,"data_stored":13008091368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1752049284},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.32600000000000001,"15min":0.41699999999999998},"min":{"1min":0.184,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":0.36499999999999999,"5min":2.5840000000000001,"15min":27.402000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.42199999999999999,"15min":0.53000000000000003},"min":{"1min":0.20999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41799999999999998,"5min":7.9820000000000002,"15min":27.538},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.57799999999999996,"15min":0.48999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.53900000000000003,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.314},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359739301,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5726572,"kb_used_data":4007332,"kb_used_omap":473,"kb_used_meta":1718758,"kb_avail":88645268,"statfs":{"total":96636764160,"available":90772754432,"internally_reserved":0,"allocated":4103507968,"data_stored":6343923052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1760008256},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:55 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.81499999999999995,"15min":0.59399999999999997},"min":{"1min":0.183,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":0.52500000000000002,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.188},{"interface":"front","average":{"1min":0.443,"5min":0.67700000000000005,"15min":0.59899999999999998},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":2.012,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:21:55 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.52800000000000002,"15min":0.501},"min":{"1min":0.188,"5min":0.185,"15min":0.184},"max":{"1min":0.81799999999999995,"5min":5.4790000000000001,"15min":15.891},"last":0.314},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.748,"15min":0.58799999999999997},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.58799999999999997,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.29899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5932528,"kb_used_data":4216680,"kb_used_omap":502,"kb_used_meta":1715337,"kb_avail":88439312,"statfs":{"total":96636764160,"available":90561855488,"internally_reserved":0,"allocated":4317880320,"data_stored":6666697993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1756506069},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,7,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.17599999999999999},"max":{"1min":0.50700000000000001,"5min":4.867,"15min":5.7729999999999997},"last":0.313},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.48299999999999998,"15min":0.49099999999999999},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.16800000000000001},"max":{"1min":1.1970000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.442,"15min":0.48599999999999999},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":4.1130000000000004,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.41799999999999998,"15min":0.47499999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61099999999999999,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4316397568,"data_stored":6666125312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4101959680,"data_stored":6343413779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8416452608,"data_stored":13007634451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:22:48.527 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:48.527 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:48.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:08.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:09.138 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:23:09.139 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:23:09.192 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2465,"stamp":"2023-10-14T01:23:07.551097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27828592357,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2098635,"num_read_kb":8164150,"num_write":2063968,"num_write_kb":8254931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87880,"ondisk_log_size":87880,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21857440,"kb_used_data":16507336,"kb_used_omap":2246,"kb_used_meta":5347833,"kb_avail":261258080,"statfs":{"total":289910292480,"available":267528273920,"internally_reserved":0,"allocated":16903512064,"data_stored":26083085149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":5476181657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,8,5,6,8,10,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22962176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5990,"num_read_kb":23832,"num_write":5922,"num_write_kb":23688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002005"},"pg_stats":[{"pgid":"2.7","version":"128'6857","reported_seq":10096,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.819141+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:23:04.819141+0000","last_peered":"2023-10-14T01:23:04.819141+0000","last_clean":"2023-10-14T01:23:04.819141+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:23:04.819141+0000","last_undegraded":"2023-10-14T01:23:04.819141+0000","last_fullsized":"2023-10-14T01:23:04.819141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'10208","reported_seq":16240,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.819173+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:23:04.819173+0000","last_peered":"2023-10-14T01:23:04.819173+0000","last_clean":"2023-10-14T01:23:04.819173+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:23:04.819173+0000","last_undegraded":"2023-10-14T01:23:04.819173+0000","last_fullsized":"2023-10-14T01:23:04.819173+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"128'6832","reported_seq":9871,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.818981+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:23:04.818981+0000","last_peered":"2023-10-14T01:23:04.818981+0000","last_clean":"2023-10-14T01:23:04.818981+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:23:04.818981+0000","last_undegraded":"2023-10-14T01:23:04.818981+0000","last_fullsized":"2023-10-14T01:23:04.818981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6832,"log_dups_size":0,"ondisk_log_size":6832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'16837","reported_seq":24889,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.819593+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:23:04.819593+0000","last_peered":"2023-10-14T01:23:04.819593+0000","last_clean":"2023-10-14T01:23:04.819593+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:23:04.819593+0000","last_undegraded":"2023-10-14T01:23:04.819593+0000","last_fullsized":"2023-10-14T01:23:04.819593+0000","mapping_epoch":12,"log_start":"96'6800","ondisk_log_start":"96'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'537506","reported_seq":1068514,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:06.551990+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:23:06.551990+0000","last_peered":"2023-10-14T01:23:06.551990+0000","last_clean":"2023-10-14T01:23:06.551990+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:23:06.551990+0000","last_undegraded":"2023-10-14T01:23:06.551990+0000","last_fullsized":"2023-10-14T01:23:06.551990+0000","mapping_epoch":15,"log_start":"127'527436","ondisk_log_start":"127'527436","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3657,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 681s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7188402176,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":532947,"num_read_kb":2100164,"num_write":532470,"num_write_kb":2129880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"128'3557","reported_seq":5264,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.818931+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:23:04.818931+0000","last_peered":"2023-10-14T01:23:04.818931+0000","last_clean":"2023-10-14T01:23:04.818931+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:23:04.818931+0000","last_undegraded":"2023-10-14T01:23:04.818931+0000","last_fullsized":"2023-10-14T01:23:04.818931+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3557,"log_dups_size":0,"ondisk_log_size":3557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'508585","reported_seq":1012927,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:05.644179+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:23:05.644179+0000","last_peered":"2023-10-14T01:23:05.644179+0000","last_clean":"2023-10-14T01:23:05.644179+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:23:05.644179+0000","last_undegraded":"2023-10-14T01:23:05.644179+0000","last_fullsized":"2023-10-14T01:23:05.644179+0000","mapping_epoch":15,"log_start":"127'498523","ondisk_log_start":"127'498523","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":488,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 86s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6827835392,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":506280,"num_read_kb":1994556,"num_write":505852,"num_write_kb":2023408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3083,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.819844+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:23:04.819844+0000","last_peered":"2023-10-14T01:23:04.819844+0000","last_clean":"2023-10-14T01:23:04.819844+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:23:04.819844+0000","last_undegraded":"2023-10-14T01:23:04.819844+0000","last_fullsized":"2023-10-14T01:23:04.819844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'530451","reported_seq":1054375,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:06.537841+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:23:06.537841+0000","last_peered":"2023-10-14T01:23:06.537841+0000","last_clean":"2023-10-14T01:23:06.537841+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:23:06.537841+0000","last_undegraded":"2023-10-14T01:23:06.537841+0000","last_fullsized":"2023-10-14T01:23:06.537841+0000","mapping_epoch":15,"log_start":"127'520428","ondisk_log_start":"127'520428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7079342080,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":525784,"num_read_kb":2071736,"num_write":525389,"num_write_kb":2101556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3233,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.819706+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:23:04.819706+0000","last_peered":"2023-10-14T01:23:04.819706+0000","last_clean":"2023-10-14T01:23:04.819706+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:23:04.819706+0000","last_undegraded":"2023-10-14T01:23:04.819706+0000","last_fullsized":"2023-10-14T01:23:04.819706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'504285","reported_seq":1002378,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:05.688635+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:23:05.688635+0000","last_peered":"2023-10-14T01:23:05.688635+0000","last_clean":"2023-10-14T01:23:05.688635+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:23:05.688635+0000","last_undegraded":"2023-10-14T01:23:05.688635+0000","last_fullsized":"2023-10-14T01:23:05.688635+0000","mapping_epoch":15,"log_start":"127'494227","ondisk_log_start":"127'494227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6732419091,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":499876,"num_read_kb":1968888,"num_write":499461,"num_write_kb":1997838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"128'10153","reported_seq":14602,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:04.819248+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:23:04.819248+0000","last_peered":"2023-10-14T01:23:04.819248+0000","last_clean":"2023-10-14T01:23:04.819248+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:23:04.819248+0000","last_undegraded":"2023-10-14T01:23:04.819248+0000","last_fullsized":"2023-10-14T01:23:04.819248+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":304,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:40.263518+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:21:40.263518+0000","last_peered":"2023-10-14T01:21:40.263518+0000","last_clean":"2023-10-14T01:21:40.263518+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:21:40.263518+0000","last_undegraded":"2023-10-14T01:21:40.263518+0000","last_fullsized":"2023-10-14T01:21:40.263518+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27827998739,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2064887,"num_read_kb":8135344,"num_write":2063172,"num_write_kb":8252682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16899698688,"data_stored":26082062374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47568,"ondisk_log_size":47568,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739306,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10058564,"kb_used_data":8251772,"kb_used_omap":1270,"kb_used_meta":1805513,"kb_avail":84313276,"statfs":{"total":96636764160,"available":86336794624,"internally_reserved":0,"allocated":8449814528,"data_stored":13040154856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1848845956},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":27.402000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.504},"min":{"1min":0.193,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.0069999999999997,"5min":7.9820000000000002,"15min":27.538},"last":0.216}]},{"osd":1,"last update":"Sat Oct 14 01:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.57799999999999996,"15min":0.48999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.53900000000000003,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.223,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.56000000000000005,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.20200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739305,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5809832,"kb_used_data":4022624,"kb_used_omap":473,"kb_used_meta":1786726,"kb_avail":88562008,"statfs":{"total":96636764160,"available":90687496192,"internally_reserved":0,"allocated":4119166976,"data_stored":6359582060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1829607488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.80900000000000005,"15min":0.57199999999999995},"min":{"1min":0.188,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":0.505,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.53600000000000003,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.52400000000000002,"15min":0.50800000000000001},"min":{"1min":0.222,"5min":0.185,"15min":0.184},"max":{"1min":1.0069999999999999,"5min":5.4790000000000001,"15min":15.891},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.74199999999999999,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.48499999999999999,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.25800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739305,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5989044,"kb_used_data":4232940,"kb_used_omap":502,"kb_used_meta":1755593,"kb_avail":88382796,"statfs":{"total":96636764160,"available":90503983104,"internally_reserved":0,"allocated":4334530560,"data_stored":6683348233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1797728213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,0,3,6,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.45900000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.093,"5min":4.867,"15min":5.7729999999999997},"last":4.085},{"interface":"front","average":{"1min":0.316,"5min":0.46300000000000002,"15min":0.47799999999999998},"min":{"1min":0.26600000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.38200000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.2189999999999999,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":3.9670000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.379,"15min":0.46000000000000002},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.40500000000000003,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4332990464,"data_stored":6682718208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4117766144,"data_stored":6359220243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8448942080,"data_stored":13040123923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:23:09.194 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:09.194 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:09.195 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:29.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:29.675 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:23:29.675 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:23:29.705 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2476,"stamp":"2023-10-14T01:23:29.554829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27859738341,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2106670,"num_read_kb":8196142,"num_write":2072001,"num_write_kb":8287063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88038,"ondisk_log_size":88038,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22069748,"kb_used_data":16571100,"kb_used_omap":2246,"kb_used_meta":5496377,"kb_avail":261045772,"statfs":{"total":289910292480,"available":267310870528,"internally_reserved":0,"allocated":16968806400,"data_stored":26148379485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":5628290713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,9,5,11,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15310848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4112,"num_read_kb":16384,"num_write":3961,"num_write_kb":15844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002018"},"pg_stats":[{"pgid":"2.7","version":"128'6885","reported_seq":10132,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.299969+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:23:25.299969+0000","last_peered":"2023-10-14T01:23:25.299969+0000","last_clean":"2023-10-14T01:23:25.299969+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:23:25.299969+0000","last_undegraded":"2023-10-14T01:23:25.299969+0000","last_fullsized":"2023-10-14T01:23:25.299969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'10250","reported_seq":16294,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.299786+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:23:25.299786+0000","last_peered":"2023-10-14T01:23:25.299786+0000","last_clean":"2023-10-14T01:23:25.299786+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:23:25.299786+0000","last_undegraded":"2023-10-14T01:23:25.299786+0000","last_fullsized":"2023-10-14T01:23:25.299786+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"128'6860","reported_seq":9907,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.299594+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:23:25.299594+0000","last_peered":"2023-10-14T01:23:25.299594+0000","last_clean":"2023-10-14T01:23:25.299594+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:23:25.299594+0000","last_undegraded":"2023-10-14T01:23:25.299594+0000","last_fullsized":"2023-10-14T01:23:25.299594+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6860,"log_dups_size":0,"ondisk_log_size":6860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'16907","reported_seq":24979,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.300206+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:23:25.300206+0000","last_peered":"2023-10-14T01:23:25.300206+0000","last_clean":"2023-10-14T01:23:25.300206+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:23:25.300206+0000","last_undegraded":"2023-10-14T01:23:25.300206+0000","last_fullsized":"2023-10-14T01:23:25.300206+0000","mapping_epoch":12,"log_start":"96'6900","ondisk_log_start":"96'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'539633","reported_seq":1072771,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:26.636464+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:23:26.636464+0000","last_peered":"2023-10-14T01:23:26.636464+0000","last_clean":"2023-10-14T01:23:26.636464+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:23:26.636464+0000","last_undegraded":"2023-10-14T01:23:26.636464+0000","last_fullsized":"2023-10-14T01:23:26.636464+0000","mapping_epoch":15,"log_start":"127'529536","ondisk_log_start":"127'529536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3725,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 701s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7196565504,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":535064,"num_read_kb":2108604,"num_write":534550,"num_write_kb":2138200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"128'3570","reported_seq":5281,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.788015+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:23:25.788015+0000","last_peered":"2023-10-14T01:23:25.788015+0000","last_clean":"2023-10-14T01:23:25.788015+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:23:25.788015+0000","last_undegraded":"2023-10-14T01:23:25.788015+0000","last_fullsized":"2023-10-14T01:23:25.788015+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'510664","reported_seq":1016922,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:25.818256+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:23:25.818256+0000","last_peered":"2023-10-14T01:23:25.818256+0000","last_clean":"2023-10-14T01:23:25.818256+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:23:25.818256+0000","last_undegraded":"2023-10-14T01:23:25.818256+0000","last_fullsized":"2023-10-14T01:23:25.818256+0000","mapping_epoch":15,"log_start":"127'500623","ondisk_log_start":"127'500623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":569,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 106s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6835671040,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":508188,"num_read_kb":2002144,"num_write":507876,"num_write_kb":2031504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3089,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:20.179149+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:23:20.179149+0000","last_peered":"2023-10-14T01:23:20.179149+0000","last_clean":"2023-10-14T01:23:20.179149+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:23:20.179149+0000","last_undegraded":"2023-10-14T01:23:20.179149+0000","last_fullsized":"2023-10-14T01:23:20.179149+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'532465","reported_seq":1058426,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:26.636637+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:23:26.636637+0000","last_peered":"2023-10-14T01:23:26.636637+0000","last_clean":"2023-10-14T01:23:26.636637+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:23:26.636637+0000","last_undegraded":"2023-10-14T01:23:26.636637+0000","last_fullsized":"2023-10-14T01:23:26.636637+0000","mapping_epoch":15,"log_start":"127'522428","ondisk_log_start":"127'522428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7087079424,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":527823,"num_read_kb":2079840,"num_write":527403,"num_write_kb":2109612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3239,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:20.179160+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:23:20.179160+0000","last_peered":"2023-10-14T01:23:20.179160+0000","last_clean":"2023-10-14T01:23:20.179160+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:23:20.179160+0000","last_undegraded":"2023-10-14T01:23:20.179160+0000","last_fullsized":"2023-10-14T01:23:20.179160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'506200","reported_seq":1006265,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.796830+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:23:25.796830+0000","last_peered":"2023-10-14T01:23:25.796830+0000","last_clean":"2023-10-14T01:23:25.796830+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:23:25.796830+0000","last_undegraded":"2023-10-14T01:23:25.796830+0000","last_fullsized":"2023-10-14T01:23:25.796830+0000","mapping_epoch":15,"log_start":"127'496127","ondisk_log_start":"127'496127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6739828755,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":501847,"num_read_kb":1976748,"num_write":501376,"num_write_kb":2005498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"128'10195","reported_seq":14656,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.300319+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:23:25.300319+0000","last_peered":"2023-10-14T01:23:25.300319+0000","last_clean":"2023-10-14T01:23:25.300319+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:23:25.300319+0000","last_undegraded":"2023-10-14T01:23:25.300319+0000","last_fullsized":"2023-10-14T01:23:25.300319+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":304,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-14T01:21:40.263518+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:21:40.263518+0000","last_peered":"2023-10-14T01:21:40.263518+0000","last_clean":"2023-10-14T01:21:40.263518+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:21:40.263518+0000","last_undegraded":"2023-10-14T01:21:40.263518+0000","last_fullsized":"2023-10-14T01:21:40.263518+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27859144723,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2072922,"num_read_kb":8167336,"num_write":2071205,"num_write_kb":8284814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16965484544,"data_stored":26147848230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40248,"ondisk_log_size":40248,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47691,"ondisk_log_size":47691,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739310,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10162476,"kb_used_data":8283492,"kb_used_omap":1270,"kb_used_meta":1877705,"kb_avail":84209364,"statfs":{"total":96636764160,"available":86230388736,"internally_reserved":0,"allocated":8482295808,"data_stored":13072636136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":1922770564},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":27.402000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.504},"min":{"1min":0.193,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.0069999999999997,"5min":7.9820000000000002,"15min":27.538},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.54000000000000004,"15min":0.47599999999999998},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.54000000000000004,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.65400000000000003,"15min":0.52500000000000002},"min":{"1min":0.20200000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.6879999999999997,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.25800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739309,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5865912,"kb_used_data":4038448,"kb_used_omap":473,"kb_used_meta":1826982,"kb_avail":88505928,"statfs":{"total":96636764160,"available":90630070272,"internally_reserved":0,"allocated":4135370752,"data_stored":6375785836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1870829632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.80900000000000005,"15min":0.57199999999999995},"min":{"1min":0.188,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":0.505,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.53600000000000003,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.63500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.52400000000000002,"15min":0.50800000000000001},"min":{"1min":0.222,"5min":0.185,"15min":0.184},"max":{"1min":1.0069999999999999,"5min":5.4790000000000001,"15min":15.891},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.74199999999999999,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.48499999999999999,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.60399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6041360,"kb_used_data":4249160,"kb_used_omap":502,"kb_used_meta":1791689,"kb_avail":88330480,"statfs":{"total":96636764160,"available":90450411520,"internally_reserved":0,"allocated":4351139840,"data_stored":6699957513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1834690517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,1,8,6,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.45900000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.093,"5min":4.867,"15min":5.7729999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.46300000000000002,"15min":0.47799999999999998},"min":{"1min":0.26600000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.38200000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.2189999999999999,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.379,"15min":0.46000000000000002},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.40500000000000003,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.372}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4349763584,"data_stored":6699491328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4133908480,"data_stored":6375362579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8481812480,"data_stored":13072994323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:23:29.707 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:29.707 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:29.708 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:30.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.451+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:23:30.459 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.451+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:23:30.459 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.451+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:23:30.460 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.455+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:23:30.460 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.455+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:23:30.461 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.455+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:23:30.461 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:30.455+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:23:46.339 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:23:46.339 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:23:46.339 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.331+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:23:46.346 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.339+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:23:46.346 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.339+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:23:46.347 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.339+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:23:46.347 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.339+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:23:46.347 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.343+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:23:46.351 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:23:46.347+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:23:49.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:50.208 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:23:50.208 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:23:50.276 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2487,"stamp":"2023-10-14T01:23:49.561075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27890454245,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2114492,"num_read_kb":8227254,"num_write":2079922,"num_write_kb":8318747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87975,"ondisk_log_size":87975,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22339748,"kb_used_data":16634568,"kb_used_omap":2246,"kb_used_meta":5702905,"kb_avail":260775772,"statfs":{"total":289910292480,"available":267034390528,"internally_reserved":0,"allocated":17033797632,"data_stored":26213359062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":5839775385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,7,5,11,2,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15679488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4075,"num_read_kb":16224,"num_write":4043,"num_write_kb":16172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002019"},"pg_stats":[{"pgid":"2.7","version":"129'6906","reported_seq":10163,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.775564+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:23:45.775564+0000","last_peered":"2023-10-14T01:23:45.775564+0000","last_clean":"2023-10-14T01:23:45.775564+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:23:45.775564+0000","last_undegraded":"2023-10-14T01:23:45.775564+0000","last_fullsized":"2023-10-14T01:23:45.775564+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'10282","reported_seq":16340,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.775389+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:23:45.775389+0000","last_peered":"2023-10-14T01:23:45.775389+0000","last_clean":"2023-10-14T01:23:45.775389+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:23:45.775389+0000","last_undegraded":"2023-10-14T01:23:45.775389+0000","last_fullsized":"2023-10-14T01:23:45.775389+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"129'6882","reported_seq":9938,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.775292+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:23:45.775292+0000","last_peered":"2023-10-14T01:23:45.775292+0000","last_clean":"2023-10-14T01:23:45.775292+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:23:45.775292+0000","last_undegraded":"2023-10-14T01:23:45.775292+0000","last_fullsized":"2023-10-14T01:23:45.775292+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"129'16959","reported_seq":25052,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.775840+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:23:45.775840+0000","last_peered":"2023-10-14T01:23:45.775840+0000","last_clean":"2023-10-14T01:23:45.775840+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:23:45.775840+0000","last_undegraded":"2023-10-14T01:23:45.775840+0000","last_fullsized":"2023-10-14T01:23:45.775840+0000","mapping_epoch":12,"log_start":"96'6900","ondisk_log_start":"96'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"129'541753","reported_seq":1076953,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:46.748581+0000","last_change":"2023-10-14T01:11:45.128800+0000","last_active":"2023-10-14T01:23:46.748581+0000","last_peered":"2023-10-14T01:23:46.748581+0000","last_clean":"2023-10-14T01:23:46.748581+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:23:46.748581+0000","last_undegraded":"2023-10-14T01:23:46.748581+0000","last_fullsized":"2023-10-14T01:23:46.748581+0000","mapping_epoch":15,"log_start":"127'531736","ondisk_log_start":"127'531736","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3809,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 721s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7204560896,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":537116,"num_read_kb":2116784,"num_write":536612,"num_write_kb":2146448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"129'3581","reported_seq":5297,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.718020+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:23:45.718020+0000","last_peered":"2023-10-14T01:23:45.718020+0000","last_clean":"2023-10-14T01:23:45.718020+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:23:45.718020+0000","last_undegraded":"2023-10-14T01:23:45.718020+0000","last_fullsized":"2023-10-14T01:23:45.718020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"129'512754","reported_seq":1020926,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:45.897357+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:23:45.897357+0000","last_peered":"2023-10-14T01:23:45.897357+0000","last_clean":"2023-10-14T01:23:45.897357+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:23:45.897357+0000","last_undegraded":"2023-10-14T01:23:45.897357+0000","last_fullsized":"2023-10-14T01:23:45.897357+0000","mapping_epoch":15,"log_start":"127'502723","ondisk_log_start":"127'502723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":696,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 126s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6843490304,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":510090,"num_read_kb":2009708,"num_write":509879,"num_write_kb":2039516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3098,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:40.679547+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:23:40.679547+0000","last_peered":"2023-10-14T01:23:40.679547+0000","last_clean":"2023-10-14T01:23:40.679547+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:23:40.679547+0000","last_undegraded":"2023-10-14T01:23:40.679547+0000","last_fullsized":"2023-10-14T01:23:40.679547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'534453","reported_seq":1062414,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:46.737947+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:23:46.737947+0000","last_peered":"2023-10-14T01:23:46.737947+0000","last_clean":"2023-10-14T01:23:46.737947+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:23:46.737947+0000","last_undegraded":"2023-10-14T01:23:46.737947+0000","last_fullsized":"2023-10-14T01:23:46.737947+0000","mapping_epoch":15,"log_start":"127'524428","ondisk_log_start":"127'524428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7094677504,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":529820,"num_read_kb":2087748,"num_write":529391,"num_write_kb":2117564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3249,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:40.665143+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:23:40.665143+0000","last_peered":"2023-10-14T01:23:40.665143+0000","last_clean":"2023-10-14T01:23:40.665143+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:23:40.665143+0000","last_undegraded":"2023-10-14T01:23:40.665143+0000","last_fullsized":"2023-10-14T01:23:40.665143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"129'508068","reported_seq":1010008,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.895695+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:23:45.895695+0000","last_peered":"2023-10-14T01:23:45.895695+0000","last_clean":"2023-10-14T01:23:45.895695+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:23:45.895695+0000","last_undegraded":"2023-10-14T01:23:45.895695+0000","last_fullsized":"2023-10-14T01:23:45.895695+0000","mapping_epoch":15,"log_start":"127'498027","ondisk_log_start":"127'498027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6747131923,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":503718,"num_read_kb":1984208,"num_write":503244,"num_write_kb":2012970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"129'10228","reported_seq":14702,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.776314+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:23:45.776314+0000","last_peered":"2023-10-14T01:23:45.776314+0000","last_clean":"2023-10-14T01:23:45.776314+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:23:45.776314+0000","last_undegraded":"2023-10-14T01:23:45.776314+0000","last_fullsized":"2023-10-14T01:23:45.776314+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":306,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:36.412231+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:23:36.412231+0000","last_peered":"2023-10-14T01:23:36.412231+0000","last_clean":"2023-10-14T01:23:36.412231+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:23:36.412231+0000","last_undegraded":"2023-10-14T01:23:36.412231+0000","last_fullsized":"2023-10-14T01:23:36.412231+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27889860627,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2080744,"num_read_kb":8198448,"num_write":2079126,"num_write_kb":8316498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17030287360,"data_stored":26212651046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40114,"ondisk_log_size":40114,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47762,"ondisk_log_size":47762,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739314,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10308192,"kb_used_data":8315220,"kb_used_omap":1270,"kb_used_meta":1991689,"kb_avail":84063648,"statfs":{"total":96636764160,"available":86081175552,"internally_reserved":0,"allocated":8514785280,"data_stored":13105121723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":2039490180},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.159},"max":{"1min":8.8360000000000003,"5min":8.8360000000000003,"15min":27.402000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.504},"min":{"1min":0.193,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.0069999999999997,"5min":7.9820000000000002,"15min":27.538},"last":0.214}]},{"osd":1,"last update":"Sat Oct 14 01:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.54000000000000004,"15min":0.47599999999999998},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.54000000000000004,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.65400000000000003,"15min":0.52500000000000002},"min":{"1min":0.20200000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.6879999999999997,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.44900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739313,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5937824,"kb_used_data":4053912,"kb_used_omap":473,"kb_used_meta":1883430,"kb_avail":88434016,"statfs":{"total":96636764160,"available":90556432384,"internally_reserved":0,"allocated":4151205888,"data_stored":6391617087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1928632384},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.80900000000000005,"15min":0.57199999999999995},"min":{"1min":0.188,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":0.505,"5min":24.248999999999999,"15min":24.248999999999999},"last":0.308},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.65300000000000002,"15min":0.57799999999999996},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.53600000000000003,"5min":17.263999999999999,"15min":17.263999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.52400000000000002,"15min":0.50800000000000001},"min":{"1min":0.222,"5min":0.185,"15min":0.184},"max":{"1min":1.0069999999999999,"5min":5.4790000000000001,"15min":15.891},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.74199999999999999,"15min":0.56100000000000005},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.48499999999999999,"5min":22.498999999999999,"15min":22.498999999999999},"last":0.34799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739313,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6093732,"kb_used_data":4265436,"kb_used_omap":502,"kb_used_meta":1827785,"kb_avail":88278108,"statfs":{"total":96636764160,"available":90396782592,"internally_reserved":0,"allocated":4367806464,"data_stored":6716620252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1871652821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,8,1,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44700000000000001,"15min":0.45900000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.093,"5min":4.867,"15min":5.7729999999999997},"last":0.38},{"interface":"front","average":{"1min":0.316,"5min":0.46300000000000002,"15min":0.47799999999999998},"min":{"1min":0.26600000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.38200000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:22:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44,"15min":0.48899999999999999},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.2189999999999999,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.379,"15min":0.46000000000000002},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.40500000000000003,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4366348288,"data_stored":6716076032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4149747712,"data_stored":6391201811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8514191360,"data_stored":13105373203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:23:50.278 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:50.278 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:50.279 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:03.746 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:03.734+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:24:03.757 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:03.746+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:24:03.758 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:03.746+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:24:10.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:10.829 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:24:10.830 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:24:10.864 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2497,"stamp":"2023-10-14T01:24:09.564350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27923001061,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2122864,"num_read_kb":8260586,"num_write":2088274,"num_write_kb":8352155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88116,"ondisk_log_size":88116,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22614452,"kb_used_data":16700892,"kb_used_omap":2246,"kb_used_meta":5911289,"kb_avail":260501068,"statfs":{"total":289910292480,"available":266753093632,"internally_reserved":0,"allocated":17101713408,"data_stored":26281274838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":6053160601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,9,6,19,11,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16506880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4258,"num_read_kb":16972,"num_write":4232,"num_write_kb":16928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001992"},"pg_stats":[{"pgid":"2.7","version":"129'6932","reported_seq":10195,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:04.997270+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:24:04.997270+0000","last_peered":"2023-10-14T01:24:04.997270+0000","last_clean":"2023-10-14T01:24:04.997270+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:24:04.997270+0000","last_undegraded":"2023-10-14T01:24:04.997270+0000","last_fullsized":"2023-10-14T01:24:04.997270+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'10322","reported_seq":16389,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:04.998785+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:24:04.998785+0000","last_peered":"2023-10-14T01:24:04.998785+0000","last_clean":"2023-10-14T01:24:04.998785+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:24:04.998785+0000","last_undegraded":"2023-10-14T01:24:04.998785+0000","last_fullsized":"2023-10-14T01:24:04.998785+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"129'6909","reported_seq":9971,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:04.990030+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:24:04.990030+0000","last_peered":"2023-10-14T01:24:04.990030+0000","last_clean":"2023-10-14T01:24:04.990030+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:24:04.990030+0000","last_undegraded":"2023-10-14T01:24:04.990030+0000","last_fullsized":"2023-10-14T01:24:04.990030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"129'17025","reported_seq":25133,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.014567+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:24:05.014567+0000","last_peered":"2023-10-14T01:24:05.014567+0000","last_clean":"2023-10-14T01:24:05.014567+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:24:05.014567+0000","last_undegraded":"2023-10-14T01:24:05.014567+0000","last_fullsized":"2023-10-14T01:24:05.014567+0000","mapping_epoch":12,"log_start":"98'7000","ondisk_log_start":"98'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"129'543973","reported_seq":1081361,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.837205+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:24:06.837205+0000","last_peered":"2023-10-14T01:24:06.837205+0000","last_clean":"2023-10-14T01:24:06.837205+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:24:06.837205+0000","last_undegraded":"2023-10-14T01:24:06.837205+0000","last_fullsized":"2023-10-14T01:24:06.837205+0000","mapping_epoch":15,"log_start":"128'533936","ondisk_log_start":"128'533936","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7213170688,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":539299,"num_read_kb":2125496,"num_write":538809,"num_write_kb":2155236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"129'3596","reported_seq":5316,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.544003+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:24:05.544003+0000","last_peered":"2023-10-14T01:24:05.544003+0000","last_clean":"2023-10-14T01:24:05.544003+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:24:05.544003+0000","last_undegraded":"2023-10-14T01:24:05.544003+0000","last_fullsized":"2023-10-14T01:24:05.544003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"129'514791","reported_seq":1024985,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:05.996516+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:24:05.996516+0000","last_peered":"2023-10-14T01:24:05.996516+0000","last_clean":"2023-10-14T01:24:05.996516+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:24:05.996516+0000","last_undegraded":"2023-10-14T01:24:05.996516+0000","last_fullsized":"2023-10-14T01:24:05.996516+0000","mapping_epoch":15,"log_start":"128'504723","ondisk_log_start":"128'504723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":771,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 146s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6851244032,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":512101,"num_read_kb":2017692,"num_write":511864,"num_write_kb":2047456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3106,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:01.135116+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:24:01.135116+0000","last_peered":"2023-10-14T01:24:01.135116+0000","last_clean":"2023-10-14T01:24:01.135116+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:24:01.135116+0000","last_undegraded":"2023-10-14T01:24:01.135116+0000","last_fullsized":"2023-10-14T01:24:01.135116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'536625","reported_seq":1066777,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.837157+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:24:06.837157+0000","last_peered":"2023-10-14T01:24:06.837157+0000","last_clean":"2023-10-14T01:24:06.837157+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:24:06.837157+0000","last_undegraded":"2023-10-14T01:24:06.837157+0000","last_fullsized":"2023-10-14T01:24:06.837157+0000","mapping_epoch":15,"log_start":"128'526528","ondisk_log_start":"128'526528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7103111168,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":532011,"num_read_kb":2096472,"num_write":531563,"num_write_kb":2126252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3257,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:01.180347+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:24:01.180347+0000","last_peered":"2023-10-14T01:24:01.180347+0000","last_clean":"2023-10-14T01:24:01.180347+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:24:01.180347+0000","last_undegraded":"2023-10-14T01:24:01.180347+0000","last_fullsized":"2023-10-14T01:24:01.180347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"129'510066","reported_seq":1013992,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.027019+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:24:06.027019+0000","last_peered":"2023-10-14T01:24:06.027019+0000","last_clean":"2023-10-14T01:24:06.027019+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:24:06.027019+0000","last_undegraded":"2023-10-14T01:24:06.027019+0000","last_fullsized":"2023-10-14T01:24:06.027019+0000","mapping_epoch":15,"log_start":"127'500027","ondisk_log_start":"127'500027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6754881555,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":505705,"num_read_kb":1992120,"num_write":505242,"num_write_kb":2020962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"129'10268","reported_seq":14751,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.027823+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:24:05.027823+0000","last_peered":"2023-10-14T01:24:05.027823+0000","last_clean":"2023-10-14T01:24:05.027823+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:24:05.027823+0000","last_undegraded":"2023-10-14T01:24:05.027823+0000","last_fullsized":"2023-10-14T01:24:05.027823+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":306,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:36.412231+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:23:36.412231+0000","last_peered":"2023-10-14T01:23:36.412231+0000","last_clean":"2023-10-14T01:23:36.412231+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:23:36.412231+0000","last_undegraded":"2023-10-14T01:23:36.412231+0000","last_fullsized":"2023-10-14T01:23:36.412231+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27922407443,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2089116,"num_read_kb":8231780,"num_write":2087478,"num_write_kb":8349906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17098539008,"data_stored":26280902694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47776,"ondisk_log_size":47776,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739318,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10455184,"kb_used_data":8348232,"kb_used_omap":1270,"kb_used_meta":2105673,"kb_avail":83916656,"statfs":{"total":96636764160,"available":85930655744,"internally_reserved":0,"allocated":8548589568,"data_stored":13138926011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":2156209796},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.45100000000000001,"15min":0.38600000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.41299999999999998,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41199999999999998,"15min":0.434},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.876,"5min":7.9820000000000002,"15min":7.9820000000000002},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.54000000000000004,"15min":0.47599999999999998},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.54000000000000004,"5min":15.845000000000001,"15min":15.845000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.65400000000000003,"15min":0.52500000000000002},"min":{"1min":0.20200000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.6879999999999997,"5min":15.805999999999999,"15min":15.805999999999999},"last":0.30399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739317,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5993920,"kb_used_data":4069752,"kb_used_omap":473,"kb_used_meta":1923686,"kb_avail":88377920,"statfs":{"total":96636764160,"available":90498990080,"internally_reserved":0,"allocated":4167426048,"data_stored":6407837247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1969854528},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,14,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.40500000000000003,"15min":0.58199999999999996},"min":{"1min":0.20799999999999999,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":3.173,"5min":4.1070000000000002,"15min":24.248999999999999},"last":0.248},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42499999999999999,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.192,"15min":0.192},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":17.263999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.52200000000000002},"min":{"1min":0.20399999999999999,"5min":0.188,"15min":0.184},"max":{"1min":4.468,"5min":4.468,"15min":15.891},"last":1.3939999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34399999999999997,"15min":0.54700000000000004},"min":{"1min":0.224,"5min":0.224,"15min":0.221},"max":{"1min":0.60399999999999998,"5min":0.76400000000000001,"15min":22.498999999999999},"last":1.5940000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739317,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6165348,"kb_used_data":4282908,"kb_used_omap":502,"kb_used_meta":1881929,"kb_avail":88206492,"statfs":{"total":96636764160,"available":90323447808,"internally_reserved":0,"allocated":4385697792,"data_stored":6734511580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1927096277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,7,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.42399999999999999,"15min":0.46999999999999997},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.085,"5min":4.085,"15min":5.7729999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.311,"5min":0.40200000000000002,"15min":0.47299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":0.45200000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.497},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.9670000000000001,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.373},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.35899999999999999,"15min":0.45200000000000001},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.41499999999999998,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4384256000,"data_stored":6733983744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4166078464,"data_stored":6407532563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8548204544,"data_stored":13139386387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:24:10.866 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:10.866 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:15.663 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:15.658+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:24:15.677 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:15.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:24:15.677 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:15.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:24:30.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:31.360 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:24:31.360 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:24:31.402 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2507,"stamp":"2023-10-14T01:24:29.569796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27955805925,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2131249,"num_read_kb":8293998,"num_write":2096707,"num_write_kb":8385887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88209,"ondisk_log_size":88209,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22892636,"kb_used_data":16768708,"kb_used_omap":2246,"kb_used_meta":6121657,"kb_avail":260222884,"statfs":{"total":289910292480,"available":266468233216,"internally_reserved":0,"allocated":17171156992,"data_stored":26350718422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":6268577433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,4,4,12,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16556032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4240,"num_read_kb":16900,"num_write":4257,"num_write_kb":17028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004139"},"pg_stats":[{"pgid":"2.7","version":"129'6954","reported_seq":10225,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:22.854650+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:24:22.854650+0000","last_peered":"2023-10-14T01:24:22.854650+0000","last_clean":"2023-10-14T01:24:22.854650+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:24:22.854650+0000","last_undegraded":"2023-10-14T01:24:22.854650+0000","last_fullsized":"2023-10-14T01:24:22.854650+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6954,"log_dups_size":0,"ondisk_log_size":6954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'10355","reported_seq":16434,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:22.862560+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:24:22.862560+0000","last_peered":"2023-10-14T01:24:22.862560+0000","last_clean":"2023-10-14T01:24:22.862560+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:24:22.862560+0000","last_undegraded":"2023-10-14T01:24:22.862560+0000","last_fullsized":"2023-10-14T01:24:22.862560+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"129'6932","reported_seq":10002,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:22.858615+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:24:22.858615+0000","last_peered":"2023-10-14T01:24:22.858615+0000","last_clean":"2023-10-14T01:24:22.858615+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:24:22.858615+0000","last_undegraded":"2023-10-14T01:24:22.858615+0000","last_fullsized":"2023-10-14T01:24:22.858615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"129'17081","reported_seq":25209,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:22.882964+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:24:22.882964+0000","last_peered":"2023-10-14T01:24:22.882964+0000","last_clean":"2023-10-14T01:24:22.882964+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:24:22.882964+0000","last_undegraded":"2023-10-14T01:24:22.882964+0000","last_fullsized":"2023-10-14T01:24:22.882964+0000","mapping_epoch":12,"log_start":"98'7000","ondisk_log_start":"98'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"129'546214","reported_seq":1085718,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.912326+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:24:26.912326+0000","last_peered":"2023-10-14T01:24:26.912326+0000","last_clean":"2023-10-14T01:24:26.912326+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:24:26.912326+0000","last_undegraded":"2023-10-14T01:24:26.912326+0000","last_fullsized":"2023-10-14T01:24:26.912326+0000","mapping_epoch":15,"log_start":"128'536136","ondisk_log_start":"128'536136","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7221919744,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":541421,"num_read_kb":2133944,"num_write":541050,"num_write_kb":2164200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"129'3613","reported_seq":5337,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:25.712732+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:24:25.712732+0000","last_peered":"2023-10-14T01:24:25.712732+0000","last_clean":"2023-10-14T01:24:25.712732+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:24:25.712732+0000","last_undegraded":"2023-10-14T01:24:25.712732+0000","last_fullsized":"2023-10-14T01:24:25.712732+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"129'516863","reported_seq":1029112,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:26.148758+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:24:26.148758+0000","last_peered":"2023-10-14T01:24:26.148758+0000","last_clean":"2023-10-14T01:24:26.148758+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:24:26.148758+0000","last_undegraded":"2023-10-14T01:24:26.148758+0000","last_fullsized":"2023-10-14T01:24:26.148758+0000","mapping_epoch":15,"log_start":"128'506823","ondisk_log_start":"128'506823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":880,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 166s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6859051008,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":514143,"num_read_kb":2025832,"num_write":513861,"num_write_kb":2055444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3114,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:21.623894+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:24:21.623894+0000","last_peered":"2023-10-14T01:24:21.623894+0000","last_clean":"2023-10-14T01:24:21.623894+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:24:21.623894+0000","last_undegraded":"2023-10-14T01:24:21.623894+0000","last_fullsized":"2023-10-14T01:24:21.623894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'538773","reported_seq":1071080,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.905885+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:24:26.905885+0000","last_peered":"2023-10-14T01:24:26.905885+0000","last_clean":"2023-10-14T01:24:26.905885+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:24:26.905885+0000","last_undegraded":"2023-10-14T01:24:26.905885+0000","last_fullsized":"2023-10-14T01:24:26.905885+0000","mapping_epoch":15,"log_start":"128'528728","ondisk_log_start":"128'528728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7111344128,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":534166,"num_read_kb":2105064,"num_write":533711,"num_write_kb":2134844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3265,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:21.630074+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:24:21.630074+0000","last_peered":"2023-10-14T01:24:21.630074+0000","last_clean":"2023-10-14T01:24:21.630074+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:24:21.630074+0000","last_undegraded":"2023-10-14T01:24:21.630074+0000","last_fullsized":"2023-10-14T01:24:21.630074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"129'512113","reported_seq":1018106,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.136797+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:24:26.136797+0000","last_peered":"2023-10-14T01:24:26.136797+0000","last_clean":"2023-10-14T01:24:26.136797+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:24:26.136797+0000","last_undegraded":"2023-10-14T01:24:26.136797+0000","last_fullsized":"2023-10-14T01:24:26.136797+0000","mapping_epoch":15,"log_start":"128'502027","ondisk_log_start":"128'502027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6762897427,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":507771,"num_read_kb":2000352,"num_write":507289,"num_write_kb":2029150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"129'10302","reported_seq":14797,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:24:22.882862+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:24:22.882862+0000","last_peered":"2023-10-14T01:24:22.882862+0000","last_clean":"2023-10-14T01:24:22.882862+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:24:22.882862+0000","last_undegraded":"2023-10-14T01:24:22.882862+0000","last_fullsized":"2023-10-14T01:24:22.882862+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":306,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:36.412231+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:23:36.412231+0000","last_peered":"2023-10-14T01:23:36.412231+0000","last_clean":"2023-10-14T01:23:36.412231+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:23:36.412231+0000","last_undegraded":"2023-10-14T01:23:36.412231+0000","last_fullsized":"2023-10-14T01:23:36.412231+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27955212307,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2097501,"num_read_kb":8265192,"num_write":2095911,"num_write_kb":8383638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17167515648,"data_stored":26349879334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40249,"ondisk_log_size":40249,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47861,"ondisk_log_size":47861,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739322,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10615036,"kb_used_data":8381748,"kb_used_omap":1270,"kb_used_meta":2232009,"kb_avail":83756804,"statfs":{"total":96636764160,"available":85766967296,"internally_reserved":0,"allocated":8582909952,"data_stored":13173246395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":2285577860},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.45100000000000001,"15min":0.38600000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.41299999999999998,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41199999999999998,"15min":0.434},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.876,"5min":7.9820000000000002,"15min":7.9820000000000002},"last":0.33500000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.48299999999999998},"min":{"1min":0.27000000000000002,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.66600000000000004,"5min":3.7509999999999999,"15min":15.845000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.503,"15min":0.53100000000000003},"min":{"1min":0.23799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.79700000000000004,"5min":7.6879999999999997,"15min":15.805999999999999},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739321,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6036592,"kb_used_data":4086696,"kb_used_omap":473,"kb_used_meta":1949414,"kb_avail":88335248,"statfs":{"total":96636764160,"available":90455293952,"internally_reserved":0,"allocated":4184776704,"data_stored":6425187903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":1996200000},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.40500000000000003,"15min":0.58199999999999996},"min":{"1min":0.20799999999999999,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":3.173,"5min":4.1070000000000002,"15min":24.248999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42499999999999999,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.192,"15min":0.192},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":17.263999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.52200000000000002},"min":{"1min":0.20399999999999999,"5min":0.188,"15min":0.184},"max":{"1min":4.468,"5min":4.468,"15min":15.891},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34399999999999997,"15min":0.54700000000000004},"min":{"1min":0.224,"5min":0.224,"15min":0.221},"max":{"1min":0.60399999999999998,"5min":0.76400000000000001,"15min":22.498999999999999},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359739321,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6241008,"kb_used_data":4300264,"kb_used_omap":502,"kb_used_meta":1940233,"kb_avail":88130832,"statfs":{"total":96636764160,"available":90245971968,"internally_reserved":0,"allocated":4403470336,"data_stored":6752284124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":1986799573},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.42399999999999999,"15min":0.46999999999999997},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.085,"5min":4.085,"15min":5.7729999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.311,"5min":0.40200000000000002,"15min":0.47299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":0.45200000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.497},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.9670000000000001,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.35899999999999999,"15min":0.45200000000000001},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.41499999999999998,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4402221056,"data_stored":6751948800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4182626304,"data_stored":6424080403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8582668288,"data_stored":13173850131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:24:31.403 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:31.404 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:41.693 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.690+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:24:41.703 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:24:41.703 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:24:41.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:24:41.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:24:41.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:24:41.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:24:41.705 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:24:41.705 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.702+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:24:41.708 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:41.706+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:24:47.017 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:24:47.018+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:24:51.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:51.952 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:24:51.953 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:24:52.017 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2519,"stamp":"2023-10-14T01:24:51.573484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27991404261,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2140603,"num_read_kb":8331206,"num_write":2105844,"num_write_kb":8422435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88219,"ondisk_log_size":88219,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22970648,"kb_used_data":16846648,"kb_used_omap":2246,"kb_used_meta":6121721,"kb_avail":260144872,"statfs":{"total":289910292480,"available":266388348928,"internally_reserved":0,"allocated":17250967552,"data_stored":26430517198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2300263,"internal_metadata":6268642969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,3,4,11,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16175104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4233,"num_read_kb":16812,"num_write":4145,"num_write_kb":16580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001652"},"pg_stats":[{"pgid":"2.7","version":"130'6980","reported_seq":10262,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:49.893897+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:24:49.893897+0000","last_peered":"2023-10-14T01:24:49.893897+0000","last_clean":"2023-10-14T01:24:49.893897+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:24:49.893897+0000","last_undegraded":"2023-10-14T01:24:49.893897+0000","last_fullsized":"2023-10-14T01:24:49.893897+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'10394","reported_seq":16489,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:49.897246+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:24:49.897246+0000","last_peered":"2023-10-14T01:24:49.897246+0000","last_clean":"2023-10-14T01:24:49.897246+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:24:49.897246+0000","last_undegraded":"2023-10-14T01:24:49.897246+0000","last_fullsized":"2023-10-14T01:24:49.897246+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"130'6958","reported_seq":10039,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:49.883816+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:24:49.883816+0000","last_peered":"2023-10-14T01:24:49.883816+0000","last_clean":"2023-10-14T01:24:49.883816+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:24:49.883816+0000","last_undegraded":"2023-10-14T01:24:49.883816+0000","last_fullsized":"2023-10-14T01:24:49.883816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6958,"log_dups_size":0,"ondisk_log_size":6958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'17146","reported_seq":25300,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:49.914662+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:24:49.914662+0000","last_peered":"2023-10-14T01:24:49.914662+0000","last_clean":"2023-10-14T01:24:49.914662+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:24:49.914662+0000","last_undegraded":"2023-10-14T01:24:49.914662+0000","last_fullsized":"2023-10-14T01:24:49.914662+0000","mapping_epoch":12,"log_start":"98'7100","ondisk_log_start":"98'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"130'548293","reported_seq":1089937,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.018193+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:24:47.018193+0000","last_peered":"2023-10-14T01:24:47.018193+0000","last_clean":"2023-10-14T01:24:47.018193+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:24:47.018193+0000","last_undegraded":"2023-10-14T01:24:47.018193+0000","last_fullsized":"2023-10-14T01:24:47.018193+0000","mapping_epoch":15,"log_start":"128'538236","ondisk_log_start":"128'538236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7230095360,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":543549,"num_read_kb":2142424,"num_write":543129,"num_write_kb":2172516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"130'3629","reported_seq":5357,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:46.496421+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:24:46.496421+0000","last_peered":"2023-10-14T01:24:46.496421+0000","last_clean":"2023-10-14T01:24:46.496421+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:24:46.496421+0000","last_undegraded":"2023-10-14T01:24:46.496421+0000","last_fullsized":"2023-10-14T01:24:46.496421+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3629,"log_dups_size":0,"ondisk_log_size":3629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"130'519465","reported_seq":1034293,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:51.313236+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:24:51.313236+0000","last_peered":"2023-10-14T01:24:51.313236+0000","last_clean":"2023-10-14T01:24:51.313236+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:24:51.313236+0000","last_undegraded":"2023-10-14T01:24:51.313236+0000","last_fullsized":"2023-10-14T01:24:51.313236+0000","mapping_epoch":15,"log_start":"128'509423","ondisk_log_start":"128'509423","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":969,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 191s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6868930560,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":516709,"num_read_kb":2036024,"num_write":516401,"num_write_kb":2065604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3126,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.218763+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:24:47.218763+0000","last_peered":"2023-10-14T01:24:47.218763+0000","last_clean":"2023-10-14T01:24:47.218763+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:24:47.218763+0000","last_undegraded":"2023-10-14T01:24:47.218763+0000","last_fullsized":"2023-10-14T01:24:47.218763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"130'540829","reported_seq":1075221,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.004460+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:24:47.004460+0000","last_peered":"2023-10-14T01:24:47.004460+0000","last_clean":"2023-10-14T01:24:47.004460+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:24:47.004460+0000","last_undegraded":"2023-10-14T01:24:47.004460+0000","last_fullsized":"2023-10-14T01:24:47.004460+0000","mapping_epoch":15,"log_start":"128'530828","ondisk_log_start":"128'530828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7119376384,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":536249,"num_read_kb":2113360,"num_write":535767,"num_write_kb":2143068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3276,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.235800+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:24:47.235800+0000","last_peered":"2023-10-14T01:24:47.235800+0000","last_clean":"2023-10-14T01:24:47.235800+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:24:47.235800+0000","last_undegraded":"2023-10-14T01:24:47.235800+0000","last_fullsized":"2023-10-14T01:24:47.235800+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"130'514575","reported_seq":1023146,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:51.311357+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:24:51.311357+0000","last_peered":"2023-10-14T01:24:51.311357+0000","last_clean":"2023-10-14T01:24:51.311357+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:24:51.311357+0000","last_undegraded":"2023-10-14T01:24:51.311357+0000","last_fullsized":"2023-10-14T01:24:51.311357+0000","mapping_epoch":15,"log_start":"128'504527","ondisk_log_start":"128'504527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6772408339,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":510348,"num_read_kb":2010592,"num_write":509751,"num_write_kb":2038998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"130'10341","reported_seq":14852,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:24:49.907831+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:24:49.907831+0000","last_peered":"2023-10-14T01:24:49.907831+0000","last_clean":"2023-10-14T01:24:49.907831+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:24:49.907831+0000","last_undegraded":"2023-10-14T01:24:49.907831+0000","last_fullsized":"2023-10-14T01:24:49.907831+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":307,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:40.884783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:23:40.884783+0000","last_peered":"2023-10-14T01:23:40.884783+0000","last_clean":"2023-10-14T01:23:40.884783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:23:40.884783+0000","last_undegraded":"2023-10-14T01:23:40.884783+0000","last_fullsized":"2023-10-14T01:23:40.884783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27990810643,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2106855,"num_read_kb":8302400,"num_write":2105048,"num_write_kb":8420186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17246601216,"data_stored":26428964902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":47972,"ondisk_log_size":47972,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739327,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10561424,"kb_used_data":8422472,"kb_used_omap":1270,"kb_used_meta":2137673,"kb_avail":83810416,"statfs":{"total":96636764160,"available":85821865984,"internally_reserved":0,"allocated":8624611328,"data_stored":13214943843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":2188977796},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.45100000000000001,"15min":0.38600000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.41299999999999998,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41199999999999998,"15min":0.434},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.876,"5min":7.9820000000000002,"15min":7.9820000000000002},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:24:05 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40799999999999997,"15min":0.48299999999999998},"min":{"1min":0.27000000000000002,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.66600000000000004,"5min":3.7509999999999999,"15min":15.845000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.503,"15min":0.53100000000000003},"min":{"1min":0.23799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.79700000000000004,"5min":7.6879999999999997,"15min":15.805999999999999},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359739326,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6114876,"kb_used_data":4106676,"kb_used_omap":473,"kb_used_meta":2007718,"kb_avail":88256964,"statfs":{"total":96636764160,"available":90375131136,"internally_reserved":0,"allocated":4205236224,"data_stored":6445643495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":2055903296},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.40500000000000003,"15min":0.58199999999999996},"min":{"1min":0.20799999999999999,"5min":0.183,"15min":0.17599999999999999},"max":{"1min":3.173,"5min":4.1070000000000002,"15min":24.248999999999999},"last":0.315},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42499999999999999,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.192,"15min":0.192},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":17.263999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.52200000000000002},"min":{"1min":0.20399999999999999,"5min":0.188,"15min":0.184},"max":{"1min":4.468,"5min":4.468,"15min":15.891},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.34399999999999997,"15min":0.54700000000000004},"min":{"1min":0.224,"5min":0.224,"15min":0.221},"max":{"1min":0.60399999999999998,"5min":0.76400000000000001,"15min":22.498999999999999},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739325,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6294348,"kb_used_data":4317500,"kb_used_omap":502,"kb_used_meta":1976329,"kb_avail":88077492,"statfs":{"total":96636764160,"available":90191351808,"internally_reserved":0,"allocated":4421120000,"data_stored":6769929860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514091,"internal_metadata":2023761877},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.42399999999999999,"15min":0.46999999999999997},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.085,"5min":4.085,"15min":5.7729999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.311,"5min":0.40200000000000002,"15min":0.47299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":0.45200000000000001,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.22600000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.45700000000000002,"15min":0.497},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.9670000000000001,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.35899999999999999,"15min":0.45200000000000001},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.41499999999999998,"5min":2.7679999999999998,"15min":4.9690000000000003},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4419166208,"data_stored":6768893952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4203118592,"data_stored":6444572691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8624316416,"data_stored":13215498259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:24:52.019 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:52.019 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:53.667 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.666+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:24:53.672 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:24:53.672 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:24:53.672 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:24:53.673 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:24:53.673 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.670+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:24:53.687 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.686+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:24:53.688 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:24:53.686+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:25:03.433 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:25:03.433+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:25:12.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:12.539 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:25:12.540 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:25:12.573 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2529,"stamp":"2023-10-14T01:25:11.577157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28023287525,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2149181,"num_read_kb":8365322,"num_write":2114036,"num_write_kb":8455203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88380,"ondisk_log_size":88380,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22591624,"kb_used_data":16911984,"kb_used_omap":2490,"kb_used_meta":5677125,"kb_avail":260523896,"statfs":{"total":289910292480,"available":266776469504,"internally_reserved":0,"allocated":17317871616,"data_stored":26497421262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2550364,"internal_metadata":5813376420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,9,19,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19902464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5378,"num_read_kb":21384,"num_write":5094,"num_write_kb":20376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002232"},"pg_stats":[{"pgid":"2.7","version":"130'7018","reported_seq":10308,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:08.602723+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:25:08.602723+0000","last_peered":"2023-10-14T01:25:08.602723+0000","last_clean":"2023-10-14T01:25:08.602723+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:25:08.602723+0000","last_undegraded":"2023-10-14T01:25:08.602723+0000","last_fullsized":"2023-10-14T01:25:08.602723+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'10451","reported_seq":16558,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:08.616401+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:25:08.616401+0000","last_peered":"2023-10-14T01:25:08.616401+0000","last_clean":"2023-10-14T01:25:08.616401+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:25:08.616401+0000","last_undegraded":"2023-10-14T01:25:08.616401+0000","last_fullsized":"2023-10-14T01:25:08.616401+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"130'6996","reported_seq":10086,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:08.595610+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:25:08.595610+0000","last_peered":"2023-10-14T01:25:08.595610+0000","last_clean":"2023-10-14T01:25:08.595610+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:25:08.595610+0000","last_undegraded":"2023-10-14T01:25:08.595610+0000","last_fullsized":"2023-10-14T01:25:08.595610+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'17241","reported_seq":25416,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:08.624058+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:25:08.624058+0000","last_peered":"2023-10-14T01:25:08.624058+0000","last_clean":"2023-10-14T01:25:08.624058+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:25:08.624058+0000","last_undegraded":"2023-10-14T01:25:08.624058+0000","last_fullsized":"2023-10-14T01:25:08.624058+0000","mapping_epoch":12,"log_start":"98'7200","ondisk_log_start":"98'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"130'550350","reported_seq":1094168,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.151693+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:25:07.151693+0000","last_peered":"2023-10-14T01:25:07.151693+0000","last_clean":"2023-10-14T01:25:07.151693+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:25:07.151693+0000","last_undegraded":"2023-10-14T01:25:07.151693+0000","last_fullsized":"2023-10-14T01:25:07.151693+0000","mapping_epoch":15,"log_start":"128'540336","ondisk_log_start":"128'540336","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7238107136,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":545726,"num_read_kb":2151092,"num_write":545186,"num_write_kb":2180744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"130'3643","reported_seq":5376,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:05.753687+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:25:05.753687+0000","last_peered":"2023-10-14T01:25:05.753687+0000","last_clean":"2023-10-14T01:25:05.753687+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:25:05.753687+0000","last_undegraded":"2023-10-14T01:25:05.753687+0000","last_fullsized":"2023-10-14T01:25:05.753687+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"130'521558","reported_seq":1038488,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:11.384479+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:25:11.384479+0000","last_peered":"2023-10-14T01:25:11.384479+0000","last_clean":"2023-10-14T01:25:11.384479+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:25:11.384479+0000","last_undegraded":"2023-10-14T01:25:11.384479+0000","last_fullsized":"2023-10-14T01:25:11.384479+0000","mapping_epoch":15,"log_start":"128'511523","ondisk_log_start":"128'511523","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1071,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 211s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6876852224,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":518801,"num_read_kb":2044336,"num_write":518424,"num_write_kb":2073696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3134,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.696292+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:25:07.696292+0000","last_peered":"2023-10-14T01:25:07.696292+0000","last_clean":"2023-10-14T01:25:07.696292+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:25:07.696292+0000","last_undegraded":"2023-10-14T01:25:07.696292+0000","last_fullsized":"2023-10-14T01:25:07.696292+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"130'542916","reported_seq":1079512,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.148830+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:25:07.148830+0000","last_peered":"2023-10-14T01:25:07.148830+0000","last_clean":"2023-10-14T01:25:07.148830+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:25:07.148830+0000","last_undegraded":"2023-10-14T01:25:07.148830+0000","last_fullsized":"2023-10-14T01:25:07.148830+0000","mapping_epoch":15,"log_start":"128'532828","ondisk_log_start":"128'532828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7127396352,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":538453,"num_read_kb":2122140,"num_write":537854,"num_write_kb":2151416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3284,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.695365+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:25:07.695365+0000","last_peered":"2023-10-14T01:25:07.695365+0000","last_clean":"2023-10-14T01:25:07.695365+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:25:07.695365+0000","last_undegraded":"2023-10-14T01:25:07.695365+0000","last_fullsized":"2023-10-14T01:25:07.695365+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"130'516600","reported_seq":1027273,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:11.416294+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:25:11.416294+0000","last_peered":"2023-10-14T01:25:11.416294+0000","last_clean":"2023-10-14T01:25:11.416294+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:25:11.416294+0000","last_undegraded":"2023-10-14T01:25:11.416294+0000","last_fullsized":"2023-10-14T01:25:11.416294+0000","mapping_epoch":15,"log_start":"128'506527","ondisk_log_start":"128'506527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6780338195,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":512453,"num_read_kb":2018948,"num_write":511776,"num_write_kb":2047098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"130'10398","reported_seq":14922,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:08.657030+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:25:08.657030+0000","last_peered":"2023-10-14T01:25:08.657030+0000","last_clean":"2023-10-14T01:25:08.657030+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:25:08.657030+0000","last_undegraded":"2023-10-14T01:25:08.657030+0000","last_fullsized":"2023-10-14T01:25:08.657030+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":307,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:40.884783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:23:40.884783+0000","last_peered":"2023-10-14T01:23:40.884783+0000","last_clean":"2023-10-14T01:23:40.884783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:23:40.884783+0000","last_undegraded":"2023-10-14T01:23:40.884783+0000","last_fullsized":"2023-10-14T01:23:40.884783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28022693907,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2115433,"num_read_kb":8336516,"num_write":2113240,"num_write_kb":8452954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17313869824,"data_stored":26496233510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40210,"ondisk_log_size":40210,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40774,"internal_metadata":0},"log_size":48071,"ondisk_log_size":48071,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739331,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10684244,"kb_used_data":8455052,"kb_used_omap":1270,"kb_used_meta":2227913,"kb_avail":83687596,"statfs":{"total":96636764160,"available":85696098304,"internally_reserved":0,"allocated":8657973248,"data_stored":13248305763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300860,"internal_metadata":2281383556},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:51 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.42199999999999999,"15min":0.38400000000000001},"min":{"1min":0.17499999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.47699999999999998,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.245},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.40999999999999998,"15min":0.432},"min":{"1min":0.221,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.45700000000000002,"5min":7.9820000000000002,"15min":7.9820000000000002},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.36399999999999999,"15min":0.47899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.51700000000000002,"5min":1.6299999999999999,"15min":15.845000000000001},"last":0.503},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.46200000000000002,"15min":0.52700000000000002},"min":{"1min":0.20399999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55600000000000005,"5min":7.6879999999999997,"15min":15.805999999999999},"last":0.48799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739330,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6167152,"kb_used_data":4122856,"kb_used_omap":473,"kb_used_meta":2043814,"kb_avail":88204688,"statfs":{"total":96636764160,"available":90321600512,"internally_reserved":0,"allocated":4221804544,"data_stored":6462211815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485312,"internal_metadata":2092865600},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,7,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.44600000000000001,"15min":0.60299999999999998},"min":{"1min":0.248,"5min":0.183,"15min":0.182},"max":{"1min":4.5860000000000003,"5min":4.5860000000000003,"15min":24.248999999999999},"last":6.1639999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.46200000000000002,"15min":0.57299999999999995},"min":{"1min":0.29899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":4.6109999999999998,"5min":4.6109999999999998,"15min":17.263999999999999},"last":6.1989999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.39600000000000002,"15min":0.46300000000000002},"min":{"1min":0.224,"5min":0.188,"15min":0.184},"max":{"1min":1.3939999999999999,"5min":4.468,"15min":5.4790000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.40400000000000003,"15min":0.56899999999999995},"min":{"1min":0.24199999999999999,"5min":0.224,"15min":0.221},"max":{"1min":4.5229999999999997,"5min":4.5229999999999997,"15min":22.498999999999999},"last":6.085}]}]},{"osd":0,"up_from":8,"seq":34359739329,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5740228,"kb_used_data":4334076,"kb_used_omap":746,"kb_used_meta":1405397,"kb_avail":88631612,"statfs":{"total":96636764160,"available":90758770688,"internally_reserved":0,"allocated":4438093824,"data_stored":6786903684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1439127264},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:59 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38700000000000001,"15min":0.45600000000000002},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.42699999999999999,"5min":4.085,"15min":5.7729999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.44700000000000001,"15min":0.46800000000000003},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":3.778,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:24:59 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.5219999999999998,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.35999999999999999,"15min":0.46100000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":4.9690000000000003},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9670,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4436131840,"data_stored":6785859584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4219699200,"data_stored":6461153299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8658038784,"data_stored":13249220627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:25:12.575 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:12.576 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:27.747 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:25:27.745+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:25:27.764 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:25:27.761+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:25:27.780 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:25:27.773+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:25:27.780 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:25:27.773+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:25:27.781 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:25:27.773+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:25:27.781 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:25:27.777+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:25:32.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:33.045 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:25:33.047 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:25:33.077 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2539,"stamp":"2023-10-14T01:25:31.580867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28055416549,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2157943,"num_read_kb":8400134,"num_write":2122313,"num_write_kb":8488311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88392,"ondisk_log_size":88392,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20487792,"kb_used_data":16977560,"kb_used_omap":2803,"kb_used_meta":3507404,"kb_avail":262627728,"statfs":{"total":289910292480,"available":268930793472,"internally_reserved":0,"allocated":17385021440,"data_stored":26564571086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":3591582298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,8,12,23,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19955712,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5510,"num_read_kb":21904,"num_write":5134,"num_write_kb":20536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001940"},"pg_stats":[{"pgid":"2.7","version":"130'7053","reported_seq":10351,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:29.254165+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:25:29.254165+0000","last_peered":"2023-10-14T01:25:29.254165+0000","last_clean":"2023-10-14T01:25:29.254165+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:25:29.254165+0000","last_undegraded":"2023-10-14T01:25:29.254165+0000","last_fullsized":"2023-10-14T01:25:29.254165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'10504","reported_seq":16623,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:29.255019+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:25:29.255019+0000","last_peered":"2023-10-14T01:25:29.255019+0000","last_clean":"2023-10-14T01:25:29.255019+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:25:29.255019+0000","last_undegraded":"2023-10-14T01:25:29.255019+0000","last_fullsized":"2023-10-14T01:25:29.255019+0000","mapping_epoch":12,"log_start":"45'500","ondisk_log_start":"45'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"130'7031","reported_seq":10129,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:29.245437+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:25:29.245437+0000","last_peered":"2023-10-14T01:25:29.245437+0000","last_clean":"2023-10-14T01:25:29.245437+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:25:29.245437+0000","last_undegraded":"2023-10-14T01:25:29.245437+0000","last_fullsized":"2023-10-14T01:25:29.245437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7031,"log_dups_size":0,"ondisk_log_size":7031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'17328","reported_seq":25523,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:29.273578+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:25:29.273578+0000","last_peered":"2023-10-14T01:25:29.273578+0000","last_clean":"2023-10-14T01:25:29.273578+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:25:29.273578+0000","last_undegraded":"2023-10-14T01:25:29.273578+0000","last_fullsized":"2023-10-14T01:25:29.273578+0000","mapping_epoch":12,"log_start":"100'7300","ondisk_log_start":"100'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"130'552538","reported_seq":1098626,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:27.245043+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:25:27.245043+0000","last_peered":"2023-10-14T01:25:27.245043+0000","last_clean":"2023-10-14T01:25:27.245043+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:25:27.245043+0000","last_undegraded":"2023-10-14T01:25:27.245043+0000","last_fullsized":"2023-10-14T01:25:27.245043+0000","mapping_epoch":15,"log_start":"129'542536","ondisk_log_start":"129'542536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7246606336,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":547998,"num_read_kb":2160100,"num_write":547374,"num_write_kb":2189496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"130'3657","reported_seq":5393,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:25.904126+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:25:25.904126+0000","last_peered":"2023-10-14T01:25:25.904126+0000","last_clean":"2023-10-14T01:25:25.904126+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:25:25.904126+0000","last_undegraded":"2023-10-14T01:25:25.904126+0000","last_fullsized":"2023-10-14T01:25:25.904126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"130'523622","reported_seq":1042713,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:31.524568+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:25:31.524568+0000","last_peered":"2023-10-14T01:25:31.524568+0000","last_clean":"2023-10-14T01:25:31.524568+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:25:31.524568+0000","last_undegraded":"2023-10-14T01:25:31.524568+0000","last_fullsized":"2023-10-14T01:25:31.524568+0000","mapping_epoch":15,"log_start":"129'513523","ondisk_log_start":"129'513523","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1154,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 231s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6884675584,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":520949,"num_read_kb":2052888,"num_write":520429,"num_write_kb":2081716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3142,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:28.176163+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:25:28.176163+0000","last_peered":"2023-10-14T01:25:28.176163+0000","last_clean":"2023-10-14T01:25:28.176163+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:25:28.176163+0000","last_undegraded":"2023-10-14T01:25:28.176163+0000","last_fullsized":"2023-10-14T01:25:28.176163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"130'544980","reported_seq":1083817,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:27.236220+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:25:27.236220+0000","last_peered":"2023-10-14T01:25:27.236220+0000","last_clean":"2023-10-14T01:25:27.236220+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:25:27.236220+0000","last_undegraded":"2023-10-14T01:25:27.236220+0000","last_fullsized":"2023-10-14T01:25:27.236220+0000","mapping_epoch":15,"log_start":"129'534928","ondisk_log_start":"129'534928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7135354880,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":540695,"num_read_kb":2131060,"num_write":539918,"num_write_kb":2159672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3292,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:28.179354+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:25:28.179354+0000","last_peered":"2023-10-14T01:25:28.179354+0000","last_clean":"2023-10-14T01:25:28.179354+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:25:28.179354+0000","last_undegraded":"2023-10-14T01:25:28.179354+0000","last_fullsized":"2023-10-14T01:25:28.179354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"130'518620","reported_seq":1031396,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:31.530246+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:25:31.530246+0000","last_peered":"2023-10-14T01:25:31.530246+0000","last_clean":"2023-10-14T01:25:31.530246+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:25:31.530246+0000","last_undegraded":"2023-10-14T01:25:31.530246+0000","last_fullsized":"2023-10-14T01:25:31.530246+0000","mapping_epoch":15,"log_start":"129'508527","ondisk_log_start":"129'508527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6788186131,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":514553,"num_read_kb":2027280,"num_write":513796,"num_write_kb":2055178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"130'10450","reported_seq":14986,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:29.279525+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:25:29.279525+0000","last_peered":"2023-10-14T01:25:29.279525+0000","last_clean":"2023-10-14T01:25:29.279525+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:25:29.279525+0000","last_undegraded":"2023-10-14T01:25:29.279525+0000","last_fullsized":"2023-10-14T01:25:29.279525+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":307,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-14T01:23:40.884783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:23:40.884783+0000","last_peered":"2023-10-14T01:23:40.884783+0000","last_clean":"2023-10-14T01:23:40.884783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:23:40.884783+0000","last_undegraded":"2023-10-14T01:23:40.884783+0000","last_fullsized":"2023-10-14T01:23:40.884783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28054822931,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2124195,"num_read_kb":8371328,"num_write":2121517,"num_write_kb":8486062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17381634048,"data_stored":26563997734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40246,"ondisk_log_size":40246,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48047,"ondisk_log_size":48047,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739335,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9405856,"kb_used_data":8488152,"kb_used_omap":1432,"kb_used_meta":916263,"kb_avail":84965984,"statfs":{"total":96636764160,"available":87005167616,"internally_reserved":0,"allocated":8691867648,"data_stored":13282200163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":938254286},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:51 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.42199999999999999,"15min":0.38400000000000001},"min":{"1min":0.17499999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.47699999999999998,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.40999999999999998,"15min":0.432},"min":{"1min":0.221,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.45700000000000002,"5min":7.9820000000000002,"15min":7.9820000000000002},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.36399999999999999,"15min":0.47899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.51700000000000002,"5min":1.6299999999999999,"15min":15.845000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.46200000000000002,"15min":0.52700000000000002},"min":{"1min":0.20399999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55600000000000005,"5min":7.6879999999999997,"15min":15.805999999999999},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359739334,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5299096,"kb_used_data":4138320,"kb_used_omap":625,"kb_used_meta":1160142,"kb_avail":89072744,"statfs":{"total":96636764160,"available":91210489856,"internally_reserved":0,"allocated":4237639680,"data_stored":6478046951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1187986348},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.44600000000000001,"15min":0.60299999999999998},"min":{"1min":0.248,"5min":0.183,"15min":0.182},"max":{"1min":4.5860000000000003,"5min":4.5860000000000003,"15min":24.248999999999999},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.46200000000000002,"15min":0.57299999999999995},"min":{"1min":0.29899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":4.6109999999999998,"5min":4.6109999999999998,"15min":17.263999999999999},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.39600000000000002,"15min":0.46300000000000002},"min":{"1min":0.224,"5min":0.188,"15min":0.184},"max":{"1min":1.3939999999999999,"5min":4.468,"15min":5.4790000000000001},"last":0.219},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.40400000000000003,"15min":0.56899999999999995},"min":{"1min":0.24199999999999999,"5min":0.224,"15min":0.221},"max":{"1min":4.5229999999999997,"5min":4.5229999999999997,"15min":22.498999999999999},"last":0.313}]}]},{"osd":0,"up_from":8,"seq":34359739333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5782840,"kb_used_data":4351088,"kb_used_omap":746,"kb_used_meta":1430997,"kb_avail":88589000,"statfs":{"total":96636764160,"available":90715136000,"internally_reserved":0,"allocated":4455514112,"data_stored":6804323972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1465341664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,7,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:59 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38700000000000001,"15min":0.45600000000000002},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.42699999999999999,"5min":4.085,"15min":5.7729999999999997},"last":0.316},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.44700000000000001,"15min":0.46800000000000003},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":3.778,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.29299999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:24:59 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.5219999999999998,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.35999999999999999,"15min":0.46100000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":4.9690000000000003},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4453556224,"data_stored":6803283968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4236185600,"data_stored":6477639699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8691892224,"data_stored":13283074067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:25:33.080 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:33.081 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:53.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:53.643 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:25:53.644 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:25:53.698 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2551,"stamp":"2023-10-14T01:25:53.587640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28092034789,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2167942,"num_read_kb":8439878,"num_write":2131717,"num_write_kb":8525927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88445,"ondisk_log_size":88445,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20584748,"kb_used_data":17048652,"kb_used_omap":2803,"kb_used_meta":3533260,"kb_avail":262530772,"statfs":{"total":289910292480,"available":268831510528,"internally_reserved":0,"allocated":17457819648,"data_stored":26637361352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":3618058842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,5,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16084992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4507,"num_read_kb":17924,"num_write":4149,"num_write_kb":16596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004851"},"pg_stats":[{"pgid":"2.7","version":"130'7089","reported_seq":10397,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.426269+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:25:51.426269+0000","last_peered":"2023-10-14T01:25:51.426269+0000","last_clean":"2023-10-14T01:25:51.426269+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:25:51.426269+0000","last_undegraded":"2023-10-14T01:25:51.426269+0000","last_fullsized":"2023-10-14T01:25:51.426269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'10558","reported_seq":16691,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.409125+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:25:51.409125+0000","last_peered":"2023-10-14T01:25:51.409125+0000","last_clean":"2023-10-14T01:25:51.409125+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:25:51.409125+0000","last_undegraded":"2023-10-14T01:25:51.409125+0000","last_fullsized":"2023-10-14T01:25:51.409125+0000","mapping_epoch":12,"log_start":"45'500","ondisk_log_start":"45'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"130'7067","reported_seq":10174,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.380503+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:25:51.380503+0000","last_peered":"2023-10-14T01:25:51.380503+0000","last_clean":"2023-10-14T01:25:51.380503+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:25:51.380503+0000","last_undegraded":"2023-10-14T01:25:51.380503+0000","last_fullsized":"2023-10-14T01:25:51.380503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'17418","reported_seq":25634,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.426287+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:25:51.426287+0000","last_peered":"2023-10-14T01:25:51.426287+0000","last_clean":"2023-10-14T01:25:51.426287+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:25:51.426287+0000","last_undegraded":"2023-10-14T01:25:51.426287+0000","last_fullsized":"2023-10-14T01:25:51.426287+0000","mapping_epoch":12,"log_start":"100'7400","ondisk_log_start":"100'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'555369","reported_seq":1104323,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:52.325781+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:25:52.325781+0000","last_peered":"2023-10-14T01:25:52.325781+0000","last_clean":"2023-10-14T01:25:52.325781+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:25:52.325781+0000","last_undegraded":"2023-10-14T01:25:52.325781+0000","last_fullsized":"2023-10-14T01:25:52.325781+0000","mapping_epoch":15,"log_start":"129'545336","ondisk_log_start":"129'545336","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7257600000,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":550858,"num_read_kb":2171460,"num_write":550205,"num_write_kb":2200820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"130'3677","reported_seq":5419,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.195321+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:25:51.195321+0000","last_peered":"2023-10-14T01:25:51.195321+0000","last_clean":"2023-10-14T01:25:51.195321+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:25:51.195321+0000","last_undegraded":"2023-10-14T01:25:51.195321+0000","last_fullsized":"2023-10-14T01:25:51.195321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'525667","reported_seq":1046948,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:51.604854+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:25:51.604854+0000","last_peered":"2023-10-14T01:25:51.604854+0000","last_clean":"2023-10-14T01:25:51.604854+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:25:51.604854+0000","last_undegraded":"2023-10-14T01:25:51.604854+0000","last_fullsized":"2023-10-14T01:25:51.604854+0000","mapping_epoch":15,"log_start":"129'515623","ondisk_log_start":"129'515623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1236,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 251s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6892404736,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":523128,"num_read_kb":2061536,"num_write":522415,"num_write_kb":2089660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3150,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:48.720218+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:25:48.720218+0000","last_peered":"2023-10-14T01:25:48.720218+0000","last_clean":"2023-10-14T01:25:48.720218+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:25:48.720218+0000","last_undegraded":"2023-10-14T01:25:48.720218+0000","last_fullsized":"2023-10-14T01:25:48.720218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'547582","reported_seq":1089224,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:52.306617+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:25:52.306617+0000","last_peered":"2023-10-14T01:25:52.306617+0000","last_clean":"2023-10-14T01:25:52.306617+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:25:52.306617+0000","last_undegraded":"2023-10-14T01:25:52.306617+0000","last_fullsized":"2023-10-14T01:25:52.306617+0000","mapping_epoch":15,"log_start":"129'537528","ondisk_log_start":"129'537528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7145398272,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":543504,"num_read_kb":2142232,"num_write":542520,"num_write_kb":2170080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3301,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:48.665400+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:25:48.665400+0000","last_peered":"2023-10-14T01:25:48.665400+0000","last_clean":"2023-10-14T01:25:48.665400+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:25:48.665400+0000","last_undegraded":"2023-10-14T01:25:48.665400+0000","last_fullsized":"2023-10-14T01:25:48.665400+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'520605","reported_seq":1035534,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.603942+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:25:51.603942+0000","last_peered":"2023-10-14T01:25:51.603942+0000","last_clean":"2023-10-14T01:25:51.603942+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:25:51.603942+0000","last_undegraded":"2023-10-14T01:25:51.603942+0000","last_fullsized":"2023-10-14T01:25:51.603942+0000","mapping_epoch":15,"log_start":"129'510527","ondisk_log_start":"129'510527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6796038163,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":516704,"num_read_kb":2035844,"num_write":515781,"num_write_kb":2063118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"130'10504","reported_seq":15053,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.434342+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:25:51.434342+0000","last_peered":"2023-10-14T01:25:51.434342+0000","last_clean":"2023-10-14T01:25:51.434342+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:25:51.434342+0000","last_undegraded":"2023-10-14T01:25:51.434342+0000","last_fullsized":"2023-10-14T01:25:51.434342+0000","mapping_epoch":12,"log_start":"45'500","ondisk_log_start":"45'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":309,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:41.578783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:25:41.578783+0000","last_peered":"2023-10-14T01:25:41.578783+0000","last_clean":"2023-10-14T01:25:41.578783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:25:41.578783+0000","last_undegraded":"2023-10-14T01:25:41.578783+0000","last_fullsized":"2023-10-14T01:25:41.578783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28091441171,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2134194,"num_read_kb":8411072,"num_write":2130921,"num_write_kb":8523678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17454252032,"data_stored":26636615718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48137,"ondisk_log_size":48137,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739339,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9376596,"kb_used_data":8521228,"kb_used_omap":1432,"kb_used_meta":853927,"kb_avail":84995244,"statfs":{"total":96636764160,"available":87035129856,"internally_reserved":0,"allocated":8725737472,"data_stored":13316069987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":874422222},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:51 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.42199999999999999,"15min":0.38400000000000001},"min":{"1min":0.17499999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.47699999999999998,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.22},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.40999999999999998,"15min":0.432},"min":{"1min":0.221,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.45700000000000002,"5min":7.9820000000000002,"15min":7.9820000000000002},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.36399999999999999,"15min":0.47899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23100000000000001,"15min":0.16700000000000001},"max":{"1min":0.51700000000000002,"5min":1.6299999999999999,"15min":15.845000000000001},"last":0.318},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.46200000000000002,"15min":0.52700000000000002},"min":{"1min":0.20399999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55600000000000005,"5min":7.6879999999999997,"15min":15.805999999999999},"last":0.26000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739338,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5362092,"kb_used_data":4154784,"kb_used_omap":625,"kb_used_meta":1206670,"kb_avail":89009748,"statfs":{"total":96636764160,"available":91145981952,"internally_reserved":0,"allocated":4254498816,"data_stored":6494902116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1235631020},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,3,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.44600000000000001,"15min":0.60299999999999998},"min":{"1min":0.248,"5min":0.183,"15min":0.182},"max":{"1min":4.5860000000000003,"5min":4.5860000000000003,"15min":24.248999999999999},"last":0.317},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.46200000000000002,"15min":0.57299999999999995},"min":{"1min":0.29899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":4.6109999999999998,"5min":4.6109999999999998,"15min":17.263999999999999},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:25:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.39600000000000002,"15min":0.46300000000000002},"min":{"1min":0.224,"5min":0.188,"15min":0.184},"max":{"1min":1.3939999999999999,"5min":4.468,"15min":5.4790000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.40400000000000003,"15min":0.56899999999999995},"min":{"1min":0.24199999999999999,"5min":0.224,"15min":0.221},"max":{"1min":4.5229999999999997,"5min":4.5229999999999997,"15min":22.498999999999999},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5846060,"kb_used_data":4372640,"kb_used_omap":746,"kb_used_meta":1472661,"kb_avail":88525780,"statfs":{"total":96636764160,"available":90650398720,"internally_reserved":0,"allocated":4477583360,"data_stored":6826389249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1508005600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:59 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38700000000000001,"15min":0.45600000000000002},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.42699999999999999,"5min":4.085,"15min":5.7729999999999997},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.44700000000000001,"15min":0.46800000000000003},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":3.778,"5min":5.2830000000000004,"15min":5.2830000000000004},"last":0.253}]},{"osd":2,"last update":"Sat Oct 14 01:24:59 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20000000000000001,"15min":0.16300000000000001},"max":{"1min":3.5219999999999998,"5min":4.1130000000000004,"15min":5.1909999999999998},"last":0.219},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.35999999999999999,"15min":0.46100000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":4.9690000000000003},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4475813888,"data_stored":6825541632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4252471296,"data_stored":6493925395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8725966848,"data_stored":13317148691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:25:53.700 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:53.700 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:13.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:14.228 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:26:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:26:14.261 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2561,"stamp":"2023-10-14T01:26:13.593123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28125060837,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2176721,"num_read_kb":8474842,"num_write":2140195,"num_write_kb":8559839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88670,"ondisk_log_size":88670,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20839364,"kb_used_data":17116332,"kb_used_omap":2803,"kb_used_meta":3720204,"kb_avail":262276156,"statfs":{"total":289910292480,"available":268570783744,"internally_reserved":0,"allocated":17527123968,"data_stored":26706661701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":3809489498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,2,5,6,6,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24727552,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6657,"num_read_kb":26508,"num_write":6353,"num_write_kb":25412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002033"},"pg_stats":[{"pgid":"2.7","version":"131'7115","reported_seq":10431,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:11.318338+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:26:11.318338+0000","last_peered":"2023-10-14T01:26:11.318338+0000","last_clean":"2023-10-14T01:26:11.318338+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:26:11.318338+0000","last_undegraded":"2023-10-14T01:26:11.318338+0000","last_fullsized":"2023-10-14T01:26:11.318338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7115,"log_dups_size":0,"ondisk_log_size":7115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'10597","reported_seq":16742,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:11.334035+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:26:11.334035+0000","last_peered":"2023-10-14T01:26:11.334035+0000","last_clean":"2023-10-14T01:26:11.334035+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:26:11.334035+0000","last_undegraded":"2023-10-14T01:26:11.334035+0000","last_fullsized":"2023-10-14T01:26:11.334035+0000","mapping_epoch":12,"log_start":"45'500","ondisk_log_start":"45'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"131'7093","reported_seq":10208,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:11.317498+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:26:11.317498+0000","last_peered":"2023-10-14T01:26:11.317498+0000","last_clean":"2023-10-14T01:26:11.317498+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:26:11.317498+0000","last_undegraded":"2023-10-14T01:26:11.317498+0000","last_fullsized":"2023-10-14T01:26:11.317498+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'17483","reported_seq":25719,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:11.337685+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:26:11.337685+0000","last_peered":"2023-10-14T01:26:11.337685+0000","last_clean":"2023-10-14T01:26:11.337685+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:26:11.337685+0000","last_undegraded":"2023-10-14T01:26:11.337685+0000","last_fullsized":"2023-10-14T01:26:11.337685+0000","mapping_epoch":12,"log_start":"100'7400","ondisk_log_start":"100'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'557542","reported_seq":1108752,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.484836+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:26:12.484836+0000","last_peered":"2023-10-14T01:26:12.484836+0000","last_clean":"2023-10-14T01:26:12.484836+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:26:12.484836+0000","last_undegraded":"2023-10-14T01:26:12.484836+0000","last_fullsized":"2023-10-14T01:26:12.484836+0000","mapping_epoch":15,"log_start":"129'547536","ondisk_log_start":"129'547536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7266140160,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":553114,"num_read_kb":2180440,"num_write":552378,"num_write_kb":2209512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"131'3687","reported_seq":5433,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:09.134910+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:26:09.134910+0000","last_peered":"2023-10-14T01:26:09.134910+0000","last_clean":"2023-10-14T01:26:09.134910+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:26:09.134910+0000","last_undegraded":"2023-10-14T01:26:09.134910+0000","last_fullsized":"2023-10-14T01:26:09.134910+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'527811","reported_seq":1051216,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:11.699506+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:26:11.699506+0000","last_peered":"2023-10-14T01:26:11.699506+0000","last_clean":"2023-10-14T01:26:11.699506+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:26:11.699506+0000","last_undegraded":"2023-10-14T01:26:11.699506+0000","last_fullsized":"2023-10-14T01:26:11.699506+0000","mapping_epoch":15,"log_start":"129'517723","ondisk_log_start":"129'517723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1300,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 272s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6900637696,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":525243,"num_read_kb":2069976,"num_write":524517,"num_write_kb":2098068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3159,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:09.140818+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:26:09.140818+0000","last_peered":"2023-10-14T01:26:09.140818+0000","last_clean":"2023-10-14T01:26:09.140818+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:26:09.140818+0000","last_undegraded":"2023-10-14T01:26:09.140818+0000","last_fullsized":"2023-10-14T01:26:09.140818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'549791","reported_seq":1093735,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.444475+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:26:12.444475+0000","last_peered":"2023-10-14T01:26:12.444475+0000","last_clean":"2023-10-14T01:26:12.444475+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:26:12.444475+0000","last_undegraded":"2023-10-14T01:26:12.444475+0000","last_fullsized":"2023-10-14T01:26:12.444475+0000","mapping_epoch":15,"log_start":"129'539728","ondisk_log_start":"129'539728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7153958912,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":545804,"num_read_kb":2151388,"num_write":544729,"num_write_kb":2178916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3310,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:09.248306+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:26:09.248306+0000","last_peered":"2023-10-14T01:26:09.248306+0000","last_clean":"2023-10-14T01:26:09.248306+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:26:09.248306+0000","last_undegraded":"2023-10-14T01:26:09.248306+0000","last_fullsized":"2023-10-14T01:26:09.248306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'522599","reported_seq":1039636,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:11.700177+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:26:11.700177+0000","last_peered":"2023-10-14T01:26:11.700177+0000","last_clean":"2023-10-14T01:26:11.700177+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:26:11.700177+0000","last_undegraded":"2023-10-14T01:26:11.700177+0000","last_fullsized":"2023-10-14T01:26:11.700177+0000","mapping_epoch":15,"log_start":"129'512527","ondisk_log_start":"129'512527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6803730451,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":518812,"num_read_kb":2044232,"num_write":517775,"num_write_kb":2071094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"131'10543","reported_seq":15104,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:11.349090+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:26:11.349090+0000","last_peered":"2023-10-14T01:26:11.349090+0000","last_clean":"2023-10-14T01:26:11.349090+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:26:11.349090+0000","last_undegraded":"2023-10-14T01:26:11.349090+0000","last_fullsized":"2023-10-14T01:26:11.349090+0000","mapping_epoch":12,"log_start":"45'500","ondisk_log_start":"45'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":309,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:41.578783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:25:41.578783+0000","last_peered":"2023-10-14T01:25:41.578783+0000","last_clean":"2023-10-14T01:25:41.578783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:25:41.578783+0000","last_undegraded":"2023-10-14T01:25:41.578783+0000","last_fullsized":"2023-10-14T01:25:41.578783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28124467219,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2142973,"num_read_kb":8446036,"num_write":2139399,"num_write_kb":8557590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17523535872,"data_stored":26705899558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40229,"ondisk_log_size":40229,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48342,"ondisk_log_size":48342,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739343,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9500628,"kb_used_data":8555020,"kb_used_omap":1432,"kb_used_meta":944167,"kb_avail":84871212,"statfs":{"total":96636764160,"available":86908121088,"internally_reserved":0,"allocated":8760340480,"data_stored":13350669024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":966827982},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:52 2023","interfaces":[{"interface":"back","average":{"1min":0.28100000000000003,"5min":0.42299999999999999,"15min":0.374},"min":{"1min":0.185,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.442,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.223},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.435},"min":{"1min":0.218,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":1.532,"5min":4.0069999999999997,"15min":7.9820000000000002},"last":0.23899999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:26:09 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.16700000000000001},"max":{"1min":11.952,"5min":11.952,"15min":15.845000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.161,"5min":0.61099999999999999,"15min":0.56799999999999995},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":12.029999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739342,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5414568,"kb_used_data":4171168,"kb_used_omap":625,"kb_used_meta":1242766,"kb_avail":88957272,"statfs":{"total":96636764160,"available":91092246528,"internally_reserved":0,"allocated":4271276032,"data_stored":6511679332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1272593324},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,3,3,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.52200000000000002,"15min":0.63700000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.23499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.496}]},{"osd":2,"last update":"Sat Oct 14 01:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35799999999999998,"15min":0.45600000000000002},"min":{"1min":0.193,"5min":0.188,"15min":0.184},"max":{"1min":0.53900000000000003,"5min":4.468,"15min":5.4790000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.47899999999999998,"15min":0.58999999999999997},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.085,"5min":6.085,"15min":22.498999999999999},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359739342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5924168,"kb_used_data":4390144,"kb_used_omap":746,"kb_used_meta":1533269,"kb_avail":88447672,"statfs":{"total":96636764160,"available":90570416128,"internally_reserved":0,"allocated":4495507456,"data_stored":6844313345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1570068192},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38500000000000001,"15min":0.433},"min":{"1min":0.17799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.72499999999999998,"5min":4.085,"15min":5.7729999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.47699999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":2.9390000000000001,"5min":3.778,"15min":5.2830000000000004},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55700000000000005,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.16300000000000001},"max":{"1min":4.2770000000000001,"5min":4.2770000000000001,"15min":5.1909999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.376,"15min":0.45800000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.8690000000000002,"5min":3.8220000000000001,"15min":4.9690000000000003},"last":0.36599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4493766656,"data_stored":6843494400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4269232128,"data_stored":6510686227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8760537088,"data_stored":13351718931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:26:14.262 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:14.263 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:26:14.784 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:26:14.808 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:20.363 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.360+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:26:20.364 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.360+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:26:20.377 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:26:20.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:26:20.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:26:20.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:26:20.378 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:26:20.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:26:20.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:26:20.379 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:26:20.380 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:26:20.380 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:26:20.380 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:26:20.376+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:26:34.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:35.268 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:26:35.269 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:26:35.331 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2571,"stamp":"2023-10-14T01:26:33.596441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28157816549,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2185548,"num_read_kb":8510022,"num_write":2148624,"num_write_kb":8593555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88653,"ondisk_log_size":88653,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21138404,"kb_used_data":17184844,"kb_used_omap":2803,"kb_used_meta":3950732,"kb_avail":261977116,"statfs":{"total":289910292480,"available":268264566784,"internally_reserved":0,"allocated":17597280256,"data_stored":26776817989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4045550170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,2,9,18,13,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24715264,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6673,"num_read_kb":26588,"num_write":6356,"num_write_kb":25424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"131'7141","reported_seq":10465,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:30.921308+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:26:30.921308+0000","last_peered":"2023-10-14T01:26:30.921308+0000","last_clean":"2023-10-14T01:26:30.921308+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:26:30.921308+0000","last_undegraded":"2023-10-14T01:26:30.921308+0000","last_fullsized":"2023-10-14T01:26:30.921308+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7141,"log_dups_size":0,"ondisk_log_size":7141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'10636","reported_seq":16793,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:30.938937+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:26:30.938937+0000","last_peered":"2023-10-14T01:26:30.938937+0000","last_clean":"2023-10-14T01:26:30.938937+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:26:30.938937+0000","last_undegraded":"2023-10-14T01:26:30.938937+0000","last_fullsized":"2023-10-14T01:26:30.938937+0000","mapping_epoch":12,"log_start":"45'600","ondisk_log_start":"45'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"131'7119","reported_seq":10242,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:30.921731+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:26:30.921731+0000","last_peered":"2023-10-14T01:26:30.921731+0000","last_clean":"2023-10-14T01:26:30.921731+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:26:30.921731+0000","last_undegraded":"2023-10-14T01:26:30.921731+0000","last_fullsized":"2023-10-14T01:26:30.921731+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7119,"log_dups_size":0,"ondisk_log_size":7119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'17548","reported_seq":25804,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:30.965256+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:26:30.965256+0000","last_peered":"2023-10-14T01:26:30.965256+0000","last_clean":"2023-10-14T01:26:30.965256+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:26:30.965256+0000","last_undegraded":"2023-10-14T01:26:30.965256+0000","last_fullsized":"2023-10-14T01:26:30.965256+0000","mapping_epoch":12,"log_start":"102'7500","ondisk_log_start":"102'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'559765","reported_seq":1113289,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:32.547889+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:26:32.547889+0000","last_peered":"2023-10-14T01:26:32.547889+0000","last_clean":"2023-10-14T01:26:32.547889+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:26:32.547889+0000","last_undegraded":"2023-10-14T01:26:32.547889+0000","last_fullsized":"2023-10-14T01:26:32.547889+0000","mapping_epoch":15,"log_start":"130'549736","ondisk_log_start":"130'549736","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7274901504,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":555427,"num_read_kb":2189656,"num_write":554601,"num_write_kb":2218404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"131'3697","reported_seq":5447,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:29.873473+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:26:29.873473+0000","last_peered":"2023-10-14T01:26:29.873473+0000","last_clean":"2023-10-14T01:26:29.873473+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:26:29.873473+0000","last_undegraded":"2023-10-14T01:26:29.873473+0000","last_fullsized":"2023-10-14T01:26:29.873473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'529876","reported_seq":1055451,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:31.849927+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:26:31.849927+0000","last_peered":"2023-10-14T01:26:31.849927+0000","last_clean":"2023-10-14T01:26:31.849927+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:26:31.849927+0000","last_undegraded":"2023-10-14T01:26:31.849927+0000","last_fullsized":"2023-10-14T01:26:31.849927+0000","mapping_epoch":15,"log_start":"130'519823","ondisk_log_start":"130'519823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1369,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 292s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6908403712,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":527400,"num_read_kb":2078568,"num_write":526533,"num_write_kb":2106132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3167,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:29.651055+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:26:29.651055+0000","last_peered":"2023-10-14T01:26:29.651055+0000","last_clean":"2023-10-14T01:26:29.651055+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:26:29.651055+0000","last_undegraded":"2023-10-14T01:26:29.651055+0000","last_fullsized":"2023-10-14T01:26:29.651055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'551964","reported_seq":1098171,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:32.539700+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:26:32.539700+0000","last_peered":"2023-10-14T01:26:32.539700+0000","last_clean":"2023-10-14T01:26:32.539700+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:26:32.539700+0000","last_undegraded":"2023-10-14T01:26:32.539700+0000","last_fullsized":"2023-10-14T01:26:32.539700+0000","mapping_epoch":15,"log_start":"130'541928","ondisk_log_start":"130'541928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7162335232,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":548065,"num_read_kb":2160392,"num_write":546902,"num_write_kb":2187608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3318,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:29.635078+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:26:29.635078+0000","last_peered":"2023-10-14T01:26:29.635078+0000","last_clean":"2023-10-14T01:26:29.635078+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:26:29.635078+0000","last_undegraded":"2023-10-14T01:26:29.635078+0000","last_fullsized":"2023-10-14T01:26:29.635078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'524616","reported_seq":1043749,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:31.862300+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:26:31.862300+0000","last_peered":"2023-10-14T01:26:31.862300+0000","last_clean":"2023-10-14T01:26:31.862300+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:26:31.862300+0000","last_undegraded":"2023-10-14T01:26:31.862300+0000","last_fullsized":"2023-10-14T01:26:31.862300+0000","mapping_epoch":15,"log_start":"130'514527","ondisk_log_start":"130'514527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6811582483,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":520908,"num_read_kb":2052600,"num_write":519792,"num_write_kb":2079162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"131'10582","reported_seq":15155,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:30.954814+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:26:30.954814+0000","last_peered":"2023-10-14T01:26:30.954814+0000","last_clean":"2023-10-14T01:26:30.954814+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:26:30.954814+0000","last_undegraded":"2023-10-14T01:26:30.954814+0000","last_fullsized":"2023-10-14T01:26:30.954814+0000","mapping_epoch":12,"log_start":"45'500","ondisk_log_start":"45'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":309,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:41.578783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:25:41.578783+0000","last_peered":"2023-10-14T01:25:41.578783+0000","last_clean":"2023-10-14T01:25:41.578783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:25:41.578783+0000","last_undegraded":"2023-10-14T01:25:41.578783+0000","last_fullsized":"2023-10-14T01:25:41.578783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28157222931,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2151800,"num_read_kb":8481216,"num_write":2147828,"num_write_kb":8591306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17592614912,"data_stored":26774978598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48347,"ondisk_log_size":48347,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739347,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9671672,"kb_used_data":8589936,"kb_used_omap":1432,"kb_used_meta":1080295,"kb_avail":84700168,"statfs":{"total":96636764160,"available":86732972032,"internally_reserved":0,"allocated":8796094464,"data_stored":13386423008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1106223054},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:52 2023","interfaces":[{"interface":"back","average":{"1min":0.28100000000000003,"5min":0.42299999999999999,"15min":0.374},"min":{"1min":0.185,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.442,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.435},"min":{"1min":0.218,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":1.532,"5min":4.0069999999999997,"15min":7.9820000000000002},"last":0.373}]},{"osd":1,"last update":"Sat Oct 14 01:26:09 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.16700000000000001},"max":{"1min":11.952,"5min":11.952,"15min":15.845000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.161,"5min":0.61099999999999999,"15min":0.56799999999999995},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":12.029999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.42399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739346,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5502780,"kb_used_data":4187188,"kb_used_omap":625,"kb_used_meta":1314958,"kb_avail":88869060,"statfs":{"total":96636764160,"available":91001917440,"internally_reserved":0,"allocated":4287680512,"data_stored":6528083812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1346517932},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,9,7,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.52200000000000002,"15min":0.63700000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.23499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35799999999999998,"15min":0.45600000000000002},"min":{"1min":0.193,"5min":0.188,"15min":0.184},"max":{"1min":0.53900000000000003,"5min":4.468,"15min":5.4790000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.47899999999999998,"15min":0.58999999999999997},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.085,"5min":6.085,"15min":22.498999999999999},"last":0.31900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739346,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5963952,"kb_used_data":4407720,"kb_used_omap":746,"kb_used_meta":1555477,"kb_avail":88407888,"statfs":{"total":96636764160,"available":90529677312,"internally_reserved":0,"allocated":4513505280,"data_stored":6862311169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1592809184},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38500000000000001,"15min":0.433},"min":{"1min":0.17799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.72499999999999998,"5min":4.085,"15min":5.7729999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.47699999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":2.9390000000000001,"5min":3.778,"15min":5.2830000000000004},"last":0.255}]},{"osd":2,"last update":"Sat Oct 14 01:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55700000000000005,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.16300000000000001},"max":{"1min":4.2770000000000001,"5min":4.2770000000000001,"15min":5.1909999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.376,"15min":0.45800000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.8690000000000002,"5min":3.8220000000000001,"15min":4.9690000000000003},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4511760384,"data_stored":6861488128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4285747200,"data_stored":6527201299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8795107328,"data_stored":13386289171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:26:35.333 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:35.333 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:37.575 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:26:37.572+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:26:55.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:55.886 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:26:55.887 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:26:55.927 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2582,"stamp":"2023-10-14T01:26:55.600625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28190879461,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2194333,"num_read_kb":8545054,"num_write":2157125,"num_write_kb":8627559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88635,"ondisk_log_size":88635,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21440056,"kb_used_data":17251872,"kb_used_omap":2803,"kb_used_meta":4185356,"kb_avail":261675464,"statfs":{"total":289910292480,"available":267955675136,"internally_reserved":0,"allocated":17665916928,"data_stored":26845454661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4285805146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,2,8,4,19,17,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16715776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4381,"num_read_kb":17456,"num_write":4278,"num_write_kb":17112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002021"},"pg_stats":[{"pgid":"2.7","version":"131'7169","reported_seq":10501,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.903328+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:26:50.903328+0000","last_peered":"2023-10-14T01:26:50.903328+0000","last_clean":"2023-10-14T01:26:50.903328+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:26:50.903328+0000","last_undegraded":"2023-10-14T01:26:50.903328+0000","last_fullsized":"2023-10-14T01:26:50.903328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7169,"log_dups_size":0,"ondisk_log_size":7169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'10678","reported_seq":16847,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.913026+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:26:50.913026+0000","last_peered":"2023-10-14T01:26:50.913026+0000","last_clean":"2023-10-14T01:26:50.913026+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:26:50.913026+0000","last_undegraded":"2023-10-14T01:26:50.913026+0000","last_fullsized":"2023-10-14T01:26:50.913026+0000","mapping_epoch":12,"log_start":"45'600","ondisk_log_start":"45'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"131'7147","reported_seq":10278,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.891336+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:26:50.891336+0000","last_peered":"2023-10-14T01:26:50.891336+0000","last_clean":"2023-10-14T01:26:50.891336+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:26:50.891336+0000","last_undegraded":"2023-10-14T01:26:50.891336+0000","last_fullsized":"2023-10-14T01:26:50.891336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7147,"log_dups_size":0,"ondisk_log_size":7147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'17618","reported_seq":25894,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.939043+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:26:50.939043+0000","last_peered":"2023-10-14T01:26:50.939043+0000","last_clean":"2023-10-14T01:26:50.939043+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:26:50.939043+0000","last_undegraded":"2023-10-14T01:26:50.939043+0000","last_fullsized":"2023-10-14T01:26:50.939043+0000","mapping_epoch":12,"log_start":"102'7600","ondisk_log_start":"102'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'561875","reported_seq":1117676,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:52.668177+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:26:52.668177+0000","last_peered":"2023-10-14T01:26:52.668177+0000","last_clean":"2023-10-14T01:26:52.668177+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:26:52.668177+0000","last_undegraded":"2023-10-14T01:26:52.668177+0000","last_fullsized":"2023-10-14T01:26:52.668177+0000","mapping_epoch":15,"log_start":"130'551836","ondisk_log_start":"130'551836","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7283113984,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":557707,"num_read_kb":2198756,"num_write":556711,"num_write_kb":2226844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"131'3711","reported_seq":5465,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:49.275025+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:26:49.275025+0000","last_peered":"2023-10-14T01:26:49.275025+0000","last_clean":"2023-10-14T01:26:49.275025+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:26:49.275025+0000","last_undegraded":"2023-10-14T01:26:49.275025+0000","last_fullsized":"2023-10-14T01:26:49.275025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3711,"log_dups_size":0,"ondisk_log_size":3711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'532050","reported_seq":1059736,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:52.015669+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:26:52.015669+0000","last_peered":"2023-10-14T01:26:52.015669+0000","last_clean":"2023-10-14T01:26:52.015669+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:26:52.015669+0000","last_undegraded":"2023-10-14T01:26:52.015669+0000","last_fullsized":"2023-10-14T01:26:52.015669+0000","mapping_epoch":15,"log_start":"130'522023","ondisk_log_start":"130'522023","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1451,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 312s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6916726784,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":529503,"num_read_kb":2086940,"num_write":528650,"num_write_kb":2114600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3176,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.096560+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:26:50.096560+0000","last_peered":"2023-10-14T01:26:50.096560+0000","last_clean":"2023-10-14T01:26:50.096560+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:26:50.096560+0000","last_undegraded":"2023-10-14T01:26:50.096560+0000","last_fullsized":"2023-10-14T01:26:50.096560+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'554139","reported_seq":1102635,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:52.659246+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:26:52.659246+0000","last_peered":"2023-10-14T01:26:52.659246+0000","last_clean":"2023-10-14T01:26:52.659246+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:26:52.659246+0000","last_undegraded":"2023-10-14T01:26:52.659246+0000","last_fullsized":"2023-10-14T01:26:52.659246+0000","mapping_epoch":15,"log_start":"130'544128","ondisk_log_start":"130'544128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7170719744,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":550352,"num_read_kb":2169516,"num_write":549077,"num_write_kb":2196308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3326,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.102667+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:26:50.102667+0000","last_peered":"2023-10-14T01:26:50.102667+0000","last_clean":"2023-10-14T01:26:50.102667+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:26:50.102667+0000","last_undegraded":"2023-10-14T01:26:50.102667+0000","last_fullsized":"2023-10-14T01:26:50.102667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'526715","reported_seq":1047960,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:52.015357+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:26:52.015357+0000","last_peered":"2023-10-14T01:26:52.015357+0000","last_clean":"2023-10-14T01:26:52.015357+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:26:52.015357+0000","last_undegraded":"2023-10-14T01:26:52.015357+0000","last_fullsized":"2023-10-14T01:26:52.015357+0000","mapping_epoch":15,"log_start":"130'516627","ondisk_log_start":"130'516627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6819725331,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":523023,"num_read_kb":2061036,"num_write":521891,"num_write_kb":2087558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"131'10624","reported_seq":15209,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:26:50.953407+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:26:50.953407+0000","last_peered":"2023-10-14T01:26:50.953407+0000","last_clean":"2023-10-14T01:26:50.953407+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:26:50.953407+0000","last_undegraded":"2023-10-14T01:26:50.953407+0000","last_fullsized":"2023-10-14T01:26:50.953407+0000","mapping_epoch":12,"log_start":"45'600","ondisk_log_start":"45'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":309,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-14T01:25:41.578783+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:25:41.578783+0000","last_peered":"2023-10-14T01:25:41.578783+0000","last_clean":"2023-10-14T01:25:41.578783+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:25:41.578783+0000","last_undegraded":"2023-10-14T01:25:41.578783+0000","last_fullsized":"2023-10-14T01:25:41.578783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28190285843,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2160585,"num_read_kb":8516248,"num_write":2156329,"num_write_kb":8625310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17662148608,"data_stored":26844512294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48371,"ondisk_log_size":48371,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739351,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9831632,"kb_used_data":8623560,"kb_used_omap":1432,"kb_used_meta":1206631,"kb_avail":84540208,"statfs":{"total":96636764160,"available":86569172992,"internally_reserved":0,"allocated":8830525440,"data_stored":13420853984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1235591118},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:52 2023","interfaces":[{"interface":"back","average":{"1min":0.28100000000000003,"5min":0.42299999999999999,"15min":0.374},"min":{"1min":0.185,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.442,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.435},"min":{"1min":0.218,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":1.532,"5min":4.0069999999999997,"15min":7.9820000000000002},"last":0.82799999999999996}]},{"osd":1,"last update":"Sat Oct 14 01:26:09 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.16700000000000001},"max":{"1min":11.952,"5min":11.952,"15min":15.845000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.161,"5min":0.61099999999999999,"15min":0.56799999999999995},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":12.029999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739350,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5555416,"kb_used_data":4203728,"kb_used_omap":625,"kb_used_meta":1351054,"kb_avail":88816424,"statfs":{"total":96636764160,"available":90948018176,"internally_reserved":0,"allocated":4304617472,"data_stored":6545020772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1383480236},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,0,3,9,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.52200000000000002,"15min":0.63700000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.23499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35799999999999998,"15min":0.45600000000000002},"min":{"1min":0.193,"5min":0.188,"15min":0.184},"max":{"1min":0.53900000000000003,"5min":4.468,"15min":5.4790000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.47899999999999998,"15min":0.58999999999999997},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.085,"5min":6.085,"15min":22.498999999999999},"last":0.65700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6053008,"kb_used_data":4424584,"kb_used_omap":746,"kb_used_meta":1627669,"kb_avail":88318832,"statfs":{"total":96636764160,"available":90438483968,"internally_reserved":0,"allocated":4530774016,"data_stored":6879579905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1666733792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,0,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38500000000000001,"15min":0.433},"min":{"1min":0.17799999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.72499999999999998,"5min":4.085,"15min":5.7729999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.47699999999999998},"min":{"1min":0.23200000000000001,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":2.9390000000000001,"5min":3.778,"15min":5.2830000000000004},"last":0.37}]},{"osd":2,"last update":"Sat Oct 14 01:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55700000000000005,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.16300000000000001},"max":{"1min":4.2770000000000001,"5min":4.2770000000000001,"15min":5.1909999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.376,"15min":0.45800000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.8690000000000002,"5min":3.8220000000000001,"15min":4.9690000000000003},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4529319936,"data_stored":6879047680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4303032320,"data_stored":6544486419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8829796352,"data_stored":13420978195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:26:55.929 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:55.929 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:15.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:16.417 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:27:16.418 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:27:16.456 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2593,"stamp":"2023-10-14T01:27:15.609802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28223065829,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2202985,"num_read_kb":8579574,"num_write":2165395,"num_write_kb":8660639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88788,"ondisk_log_size":88788,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21681848,"kb_used_data":17317784,"kb_used_omap":2803,"kb_used_meta":4361228,"kb_avail":261433672,"statfs":{"total":289910292480,"available":267708080128,"internally_reserved":0,"allocated":17733410816,"data_stored":26912940521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4465898074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,15,7,8,14,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16060416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4231,"num_read_kb":16880,"num_write":4127,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001556"},"pg_stats":[{"pgid":"2.7","version":"131'7197","reported_seq":10538,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.575571+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:27:10.575571+0000","last_peered":"2023-10-14T01:27:10.575571+0000","last_clean":"2023-10-14T01:27:10.575571+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:27:10.575571+0000","last_undegraded":"2023-10-14T01:27:10.575571+0000","last_fullsized":"2023-10-14T01:27:10.575571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7197,"log_dups_size":0,"ondisk_log_size":7197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'10720","reported_seq":16902,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.575359+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:27:10.575359+0000","last_peered":"2023-10-14T01:27:10.575359+0000","last_clean":"2023-10-14T01:27:10.575359+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:27:10.575359+0000","last_undegraded":"2023-10-14T01:27:10.575359+0000","last_fullsized":"2023-10-14T01:27:10.575359+0000","mapping_epoch":12,"log_start":"47'700","ondisk_log_start":"47'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"131'7175","reported_seq":10316,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.575650+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:27:10.575650+0000","last_peered":"2023-10-14T01:27:10.575650+0000","last_clean":"2023-10-14T01:27:10.575650+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:27:10.575650+0000","last_undegraded":"2023-10-14T01:27:10.575650+0000","last_fullsized":"2023-10-14T01:27:10.575650+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7175,"log_dups_size":0,"ondisk_log_size":7175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'17688","reported_seq":25986,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.577358+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:27:10.577358+0000","last_peered":"2023-10-14T01:27:10.577358+0000","last_clean":"2023-10-14T01:27:10.577358+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:27:10.577358+0000","last_undegraded":"2023-10-14T01:27:10.577358+0000","last_fullsized":"2023-10-14T01:27:10.577358+0000","mapping_epoch":12,"log_start":"102'7600","ondisk_log_start":"102'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'563953","reported_seq":1121996,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:12.756859+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:27:12.756859+0000","last_peered":"2023-10-14T01:27:12.756859+0000","last_clean":"2023-10-14T01:27:12.756859+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:27:12.756859+0000","last_undegraded":"2023-10-14T01:27:12.756859+0000","last_fullsized":"2023-10-14T01:27:12.756859+0000","mapping_epoch":15,"log_start":"130'553936","ondisk_log_start":"130'553936","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7291219968,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":559946,"num_read_kb":2207676,"num_write":558789,"num_write_kb":2235156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'3722","reported_seq":5482,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:11.435119+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:27:11.435119+0000","last_peered":"2023-10-14T01:27:11.435119+0000","last_clean":"2023-10-14T01:27:11.435119+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:27:11.435119+0000","last_undegraded":"2023-10-14T01:27:11.435119+0000","last_fullsized":"2023-10-14T01:27:11.435119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'534186","reported_seq":1064028,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:12.139581+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:27:12.139581+0000","last_peered":"2023-10-14T01:27:12.139581+0000","last_clean":"2023-10-14T01:27:12.139581+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:27:12.139581+0000","last_undegraded":"2023-10-14T01:27:12.139581+0000","last_fullsized":"2023-10-14T01:27:12.139581+0000","mapping_epoch":15,"log_start":"130'524123","ondisk_log_start":"130'524123","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1539,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 332s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6924853248,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":531646,"num_read_kb":2095496,"num_write":530724,"num_write_kb":2122896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3184,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.575039+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:27:10.575039+0000","last_peered":"2023-10-14T01:27:10.575039+0000","last_clean":"2023-10-14T01:27:10.575039+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:27:10.575039+0000","last_undegraded":"2023-10-14T01:27:10.575039+0000","last_fullsized":"2023-10-14T01:27:10.575039+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'556226","reported_seq":1106913,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:12.752520+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:27:12.752520+0000","last_peered":"2023-10-14T01:27:12.752520+0000","last_clean":"2023-10-14T01:27:12.752520+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:27:12.752520+0000","last_undegraded":"2023-10-14T01:27:12.752520+0000","last_fullsized":"2023-10-14T01:27:12.752520+0000","mapping_epoch":15,"log_start":"130'546128","ondisk_log_start":"130'546128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7178756096,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":552542,"num_read_kb":2178252,"num_write":551164,"num_write_kb":2204656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3334,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.585372+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:27:10.585372+0000","last_peered":"2023-10-14T01:27:10.585372+0000","last_clean":"2023-10-14T01:27:10.585372+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:27:10.585372+0000","last_undegraded":"2023-10-14T01:27:10.585372+0000","last_fullsized":"2023-10-14T01:27:10.585372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'528746","reported_seq":1052073,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:12.139756+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:27:12.139756+0000","last_peered":"2023-10-14T01:27:12.139756+0000","last_clean":"2023-10-14T01:27:12.139756+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:27:12.139756+0000","last_undegraded":"2023-10-14T01:27:12.139756+0000","last_fullsized":"2023-10-14T01:27:12.139756+0000","mapping_epoch":15,"log_start":"130'518727","ondisk_log_start":"130'518727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6827642899,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":525103,"num_read_kb":2069344,"num_write":523922,"num_write_kb":2095682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"131'10666","reported_seq":15265,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:27:10.575799+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:27:10.575799+0000","last_peered":"2023-10-14T01:27:10.575799+0000","last_clean":"2023-10-14T01:27:10.575799+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:27:10.575799+0000","last_undegraded":"2023-10-14T01:27:10.575799+0000","last_fullsized":"2023-10-14T01:27:10.575799+0000","mapping_epoch":12,"log_start":"45'600","ondisk_log_start":"45'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":310,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.432624+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:25:51.432624+0000","last_peered":"2023-10-14T01:25:51.432624+0000","last_clean":"2023-10-14T01:25:51.432624+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:25:51.432624+0000","last_undegraded":"2023-10-14T01:25:51.432624+0000","last_fullsized":"2023-10-14T01:25:51.432624+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28222472211,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2169237,"num_read_kb":8550768,"num_write":2164599,"num_write_kb":8658390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17729957888,"data_stored":26912321574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48492,"ondisk_log_size":48492,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739355,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9964536,"kb_used_data":8656368,"kb_used_omap":1432,"kb_used_meta":1306727,"kb_avail":84407304,"statfs":{"total":96636764160,"available":86433079296,"internally_reserved":0,"allocated":8864120832,"data_stored":13454449376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1338089422},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":0.41699999999999998,"15min":0.36399999999999999},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.38800000000000001,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.36499999999999999,"15min":0.437},"min":{"1min":0.215,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.82799999999999996,"5min":4.0069999999999997,"15min":7.9820000000000002},"last":0.23799999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:26:09 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.51800000000000002,"15min":0.52400000000000002},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.16700000000000001},"max":{"1min":11.952,"5min":11.952,"15min":15.845000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.161,"5min":0.61099999999999999,"15min":0.56799999999999995},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":12.029999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.308}]}]},{"osd":1,"up_from":8,"seq":34359739354,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5611492,"kb_used_data":4220120,"kb_used_omap":625,"kb_used_meta":1390734,"kb_avail":88760348,"statfs":{"total":96636764160,"available":90890596352,"internally_reserved":0,"allocated":4321402880,"data_stored":6561802166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1424112556},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,4,6,14,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.52800000000000002,"15min":0.63800000000000001},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.72299999999999998,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.497},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.57199999999999995,"15min":0.58799999999999997},"min":{"1min":0.27800000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.73999999999999999,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:27:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.40400000000000003,"15min":0.46999999999999997},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.185},"max":{"1min":4.8319999999999999,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.48099999999999998,"15min":0.59099999999999997},"min":{"1min":0.23799999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65700000000000003,"5min":6.085,"15min":22.498999999999999},"last":0.45900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6105820,"kb_used_data":4441296,"kb_used_omap":746,"kb_used_meta":1663765,"kb_avail":88266020,"statfs":{"total":96636764160,"available":90384404480,"internally_reserved":0,"allocated":4547887104,"data_stored":6896688979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1703696096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":0.58399999999999996,"15min":0.48499999999999999},"min":{"1min":0.193,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.023,"5min":16.023,"15min":16.023},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.17899999999999999},"max":{"1min":16.065000000000001,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16300000000000001},"max":{"1min":0.52800000000000002,"5min":4.2770000000000001,"15min":4.2770000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.373,"15min":0.442},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.44400000000000001,"5min":3.8220000000000001,"15min":4.8049999999999997},"last":0.44600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4546383872,"data_stored":6896111616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4319825920,"data_stored":6561280019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8863748096,"data_stored":13454929939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:27:16.458 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:16.458 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:17.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:27:17.251+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:27:17.253 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:27:17.251+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:27:17.253 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:27:17.251+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:27:36.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:37.031 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:27:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:27:37.069 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2603,"stamp":"2023-10-14T01:27:35.613070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28255268581,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2211382,"num_read_kb":8613054,"num_write":2173656,"num_write_kb":8693683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88894,"ondisk_log_size":88894,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21936720,"kb_used_data":17383864,"kb_used_omap":2803,"kb_used_meta":4550028,"kb_avail":261178800,"statfs":{"total":289910292480,"available":267447091200,"internally_reserved":0,"allocated":17801076736,"data_stored":26980602427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4659229274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,8,8,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16359424,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4291,"num_read_kb":17116,"num_write":4182,"num_write_kb":16728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001920"},"pg_stats":[{"pgid":"2.7","version":"132'7232","reported_seq":10581,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.006592+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:27:32.006592+0000","last_peered":"2023-10-14T01:27:32.006592+0000","last_clean":"2023-10-14T01:27:32.006592+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:27:32.006592+0000","last_undegraded":"2023-10-14T01:27:32.006592+0000","last_fullsized":"2023-10-14T01:27:32.006592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'10772","reported_seq":16966,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.013803+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:27:32.013803+0000","last_peered":"2023-10-14T01:27:32.013803+0000","last_clean":"2023-10-14T01:27:32.013803+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:27:32.013803+0000","last_undegraded":"2023-10-14T01:27:32.013803+0000","last_fullsized":"2023-10-14T01:27:32.013803+0000","mapping_epoch":12,"log_start":"47'700","ondisk_log_start":"47'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"132'7210","reported_seq":10359,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.995352+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:27:31.995352+0000","last_peered":"2023-10-14T01:27:31.995352+0000","last_clean":"2023-10-14T01:27:31.995352+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:27:31.995352+0000","last_undegraded":"2023-10-14T01:27:31.995352+0000","last_fullsized":"2023-10-14T01:27:31.995352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7210,"log_dups_size":0,"ondisk_log_size":7210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'17774","reported_seq":26092,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.042914+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:27:32.042914+0000","last_peered":"2023-10-14T01:27:32.042914+0000","last_clean":"2023-10-14T01:27:32.042914+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:27:32.042914+0000","last_undegraded":"2023-10-14T01:27:32.042914+0000","last_fullsized":"2023-10-14T01:27:32.042914+0000","mapping_epoch":12,"log_start":"103'7700","ondisk_log_start":"103'7700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'566069","reported_seq":1126237,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.846685+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:27:32.846685+0000","last_peered":"2023-10-14T01:27:32.846685+0000","last_clean":"2023-10-14T01:27:32.846685+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:27:32.846685+0000","last_undegraded":"2023-10-14T01:27:32.846685+0000","last_fullsized":"2023-10-14T01:27:32.846685+0000","mapping_epoch":15,"log_start":"131'556036","ondisk_log_start":"131'556036","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7299506176,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":562072,"num_read_kb":2216144,"num_write":560905,"num_write_kb":2243620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'3740","reported_seq":5504,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.058907+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:27:31.058907+0000","last_peered":"2023-10-14T01:27:31.058907+0000","last_clean":"2023-10-14T01:27:31.058907+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:27:31.058907+0000","last_undegraded":"2023-10-14T01:27:31.058907+0000","last_fullsized":"2023-10-14T01:27:31.058907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'536302","reported_seq":1068161,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:32.193854+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:27:32.193854+0000","last_peered":"2023-10-14T01:27:32.193854+0000","last_clean":"2023-10-14T01:27:32.193854+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:27:32.193854+0000","last_undegraded":"2023-10-14T01:27:32.193854+0000","last_fullsized":"2023-10-14T01:27:32.193854+0000","mapping_epoch":15,"log_start":"131'526223","ondisk_log_start":"131'526223","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1635,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 352s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6932783104,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":533655,"num_read_kb":2103516,"num_write":532772,"num_write_kb":2131088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3193,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.060306+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:27:31.060306+0000","last_peered":"2023-10-14T01:27:31.060306+0000","last_clean":"2023-10-14T01:27:31.060306+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:27:31.060306+0000","last_undegraded":"2023-10-14T01:27:31.060306+0000","last_fullsized":"2023-10-14T01:27:31.060306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'558310","reported_seq":1111118,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.845814+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:27:32.845814+0000","last_peered":"2023-10-14T01:27:32.845814+0000","last_clean":"2023-10-14T01:27:32.845814+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:27:32.845814+0000","last_undegraded":"2023-10-14T01:27:32.845814+0000","last_fullsized":"2023-10-14T01:27:32.845814+0000","mapping_epoch":15,"log_start":"131'548228","ondisk_log_start":"131'548228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7186874368,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":554667,"num_read_kb":2186728,"num_write":553248,"num_write_kb":2212992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3343,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.074949+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:27:31.074949+0000","last_peered":"2023-10-14T01:27:31.074949+0000","last_clean":"2023-10-14T01:27:31.074949+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:27:31.074949+0000","last_undegraded":"2023-10-14T01:27:31.074949+0000","last_fullsized":"2023-10-14T01:27:31.074949+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'530759","reported_seq":1056225,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.190488+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:27:32.190488+0000","last_peered":"2023-10-14T01:27:32.190488+0000","last_clean":"2023-10-14T01:27:32.190488+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:27:32.190488+0000","last_undegraded":"2023-10-14T01:27:32.190488+0000","last_fullsized":"2023-10-14T01:27:32.190488+0000","mapping_epoch":15,"log_start":"131'520727","ondisk_log_start":"131'520727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6835511315,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":527240,"num_read_kb":2077860,"num_write":525935,"num_write_kb":2103734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"132'10717","reported_seq":15328,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.037615+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:27:32.037615+0000","last_peered":"2023-10-14T01:27:32.037615+0000","last_clean":"2023-10-14T01:27:32.037615+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:27:32.037615+0000","last_undegraded":"2023-10-14T01:27:32.037615+0000","last_fullsized":"2023-10-14T01:27:32.037615+0000","mapping_epoch":12,"log_start":"47'700","ondisk_log_start":"47'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":310,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-14T01:25:51.432624+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:25:51.432624+0000","last_peered":"2023-10-14T01:25:51.432624+0000","last_clean":"2023-10-14T01:25:51.432624+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:25:51.432624+0000","last_undegraded":"2023-10-14T01:25:51.432624+0000","last_fullsized":"2023-10-14T01:25:51.432624+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28254674963,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2177634,"num_read_kb":8584248,"num_write":2172860,"num_write_kb":8691434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17797636096,"data_stored":26979999782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40226,"ondisk_log_size":40226,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48569,"ondisk_log_size":48569,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10087624,"kb_used_data":8689216,"kb_used_omap":1432,"kb_used_meta":1396967,"kb_avail":84284216,"statfs":{"total":96636764160,"available":86307037184,"internally_reserved":0,"allocated":8897757184,"data_stored":13488081714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1430495182},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":0.41699999999999998,"15min":0.36399999999999999},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.38800000000000001,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.36499999999999999,"15min":0.437},"min":{"1min":0.215,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.82799999999999996,"5min":4.0069999999999997,"15min":7.9820000000000002},"last":0.21099999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:27:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":4.1929999999999996,"5min":11.952,"15min":15.845000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64700000000000002,"15min":0.57599999999999996},"min":{"1min":0.26300000000000001,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.2089999999999996,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.35299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739358,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5686172,"kb_used_data":4236500,"kb_used_omap":625,"kb_used_meta":1449038,"kb_avail":88685668,"statfs":{"total":96636764160,"available":90814124032,"internally_reserved":0,"allocated":4338176000,"data_stored":6578575286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1483815852},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.52800000000000002,"15min":0.63800000000000001},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.72299999999999998,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.495},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.57199999999999995,"15min":0.58799999999999997},"min":{"1min":0.27800000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.73999999999999999,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:27:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.40400000000000003,"15min":0.46999999999999997},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.185},"max":{"1min":4.8319999999999999,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.48099999999999998,"15min":0.59099999999999997},"min":{"1min":0.23799999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65700000000000003,"5min":6.085,"15min":22.498999999999999},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739358,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6162924,"kb_used_data":4458148,"kb_used_omap":746,"kb_used_meta":1704021,"kb_avail":88208916,"statfs":{"total":96636764160,"available":90325929984,"internally_reserved":0,"allocated":4565143552,"data_stored":6913945427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1744918240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":0.58399999999999996,"15min":0.48499999999999999},"min":{"1min":0.193,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.023,"5min":16.023,"15min":16.023},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.17899999999999999},"max":{"1min":16.065000000000001,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.37}]},{"osd":2,"last update":"Sat Oct 14 01:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16300000000000001},"max":{"1min":0.52800000000000002,"5min":4.2770000000000001,"15min":4.2770000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.373,"15min":0.442},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.44400000000000001,"5min":3.8220000000000001,"15min":4.8049999999999997},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4563582976,"data_stored":6913310720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4336459776,"data_stored":6577913875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8897593344,"data_stored":13488775187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:27:37.071 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:37.072 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:57.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:57.557 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:27:57.558 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:27:57.586 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2613,"stamp":"2023-10-14T01:27:55.616408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28288134885,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2219791,"num_read_kb":8646570,"num_write":2182063,"num_write_kb":8727311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88936,"ondisk_log_size":88936,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22138856,"kb_used_data":17450828,"kb_used_omap":2803,"kb_used_meta":4685196,"kb_avail":260976664,"statfs":{"total":289910292480,"available":267240103936,"internally_reserved":0,"allocated":17869647872,"data_stored":27049173563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4797641306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,6,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16605184,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4295,"num_read_kb":17120,"num_write":4253,"num_write_kb":17012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002074"},"pg_stats":[{"pgid":"2.7","version":"132'7256","reported_seq":10613,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:51.535112+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:27:51.535112+0000","last_peered":"2023-10-14T01:27:51.535112+0000","last_clean":"2023-10-14T01:27:51.535112+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:27:51.535112+0000","last_undegraded":"2023-10-14T01:27:51.535112+0000","last_fullsized":"2023-10-14T01:27:51.535112+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7256,"log_dups_size":0,"ondisk_log_size":7256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'10808","reported_seq":17015,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:51.535530+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:27:51.535530+0000","last_peered":"2023-10-14T01:27:51.535530+0000","last_clean":"2023-10-14T01:27:51.535530+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:27:51.535530+0000","last_undegraded":"2023-10-14T01:27:51.535530+0000","last_fullsized":"2023-10-14T01:27:51.535530+0000","mapping_epoch":12,"log_start":"47'800","ondisk_log_start":"47'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"132'7234","reported_seq":10391,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:51.534889+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:27:51.534889+0000","last_peered":"2023-10-14T01:27:51.534889+0000","last_clean":"2023-10-14T01:27:51.534889+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:27:51.534889+0000","last_undegraded":"2023-10-14T01:27:51.534889+0000","last_fullsized":"2023-10-14T01:27:51.534889+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7234,"log_dups_size":0,"ondisk_log_size":7234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'17834","reported_seq":26172,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:51.535647+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:27:51.535647+0000","last_peered":"2023-10-14T01:27:51.535647+0000","last_clean":"2023-10-14T01:27:51.535647+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:27:51.535647+0000","last_undegraded":"2023-10-14T01:27:51.535647+0000","last_fullsized":"2023-10-14T01:27:51.535647+0000","mapping_epoch":12,"log_start":"104'7800","ondisk_log_start":"104'7800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'568320","reported_seq":1130697,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.961856+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:27:52.961856+0000","last_peered":"2023-10-14T01:27:52.961856+0000","last_clean":"2023-10-14T01:27:52.961856+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:27:52.961856+0000","last_undegraded":"2023-10-14T01:27:52.961856+0000","last_fullsized":"2023-10-14T01:27:52.961856+0000","mapping_epoch":15,"log_start":"131'558236","ondisk_log_start":"131'558236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7308296192,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":564278,"num_read_kb":2224932,"num_write":563156,"num_write_kb":2252624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'3752","reported_seq":5520,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:51.535788+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:27:51.535788+0000","last_peered":"2023-10-14T01:27:51.535788+0000","last_clean":"2023-10-14T01:27:51.535788+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:27:51.535788+0000","last_undegraded":"2023-10-14T01:27:51.535788+0000","last_fullsized":"2023-10-14T01:27:51.535788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'538385","reported_seq":1072324,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:52.391985+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:27:52.391985+0000","last_peered":"2023-10-14T01:27:52.391985+0000","last_clean":"2023-10-14T01:27:52.391985+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:27:52.391985+0000","last_undegraded":"2023-10-14T01:27:52.391985+0000","last_fullsized":"2023-10-14T01:27:52.391985+0000","mapping_epoch":15,"log_start":"131'528323","ondisk_log_start":"131'528323","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1698,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 372s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6940770304,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":535726,"num_read_kb":2111768,"num_write":534812,"num_write_kb":2139248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3199,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:46.427692+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:27:46.427692+0000","last_peered":"2023-10-14T01:27:46.427692+0000","last_clean":"2023-10-14T01:27:46.427692+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:27:46.427692+0000","last_undegraded":"2023-10-14T01:27:46.427692+0000","last_fullsized":"2023-10-14T01:27:46.427692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'560398","reported_seq":1115278,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.955829+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:27:52.955829+0000","last_peered":"2023-10-14T01:27:52.955829+0000","last_clean":"2023-10-14T01:27:52.955829+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:27:52.955829+0000","last_undegraded":"2023-10-14T01:27:52.955829+0000","last_fullsized":"2023-10-14T01:27:52.955829+0000","mapping_epoch":15,"log_start":"131'550328","ondisk_log_start":"131'550328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7195004928,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":556735,"num_read_kb":2194976,"num_write":555336,"num_write_kb":2221344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3350,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:46.440990+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:27:46.440990+0000","last_peered":"2023-10-14T01:27:46.440990+0000","last_clean":"2023-10-14T01:27:46.440990+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:27:46.440990+0000","last_undegraded":"2023-10-14T01:27:46.440990+0000","last_fullsized":"2023-10-14T01:27:46.440990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'532787","reported_seq":1060315,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.379510+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:27:52.379510+0000","last_peered":"2023-10-14T01:27:52.379510+0000","last_clean":"2023-10-14T01:27:52.379510+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:27:52.379510+0000","last_undegraded":"2023-10-14T01:27:52.379510+0000","last_fullsized":"2023-10-14T01:27:52.379510+0000","mapping_epoch":15,"log_start":"131'522727","ondisk_log_start":"131'522727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6843469843,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":529304,"num_read_kb":2086088,"num_write":527963,"num_write_kb":2111846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"132'10753","reported_seq":15376,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:51.535253+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:27:51.535253+0000","last_peered":"2023-10-14T01:27:51.535253+0000","last_clean":"2023-10-14T01:27:51.535253+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:27:51.535253+0000","last_undegraded":"2023-10-14T01:27:51.535253+0000","last_fullsized":"2023-10-14T01:27:51.535253+0000","mapping_epoch":12,"log_start":"47'700","ondisk_log_start":"47'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":311,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:11.447164+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:27:11.447164+0000","last_peered":"2023-10-14T01:27:11.447164+0000","last_clean":"2023-10-14T01:27:11.447164+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:27:11.447164+0000","last_undegraded":"2023-10-14T01:27:11.447164+0000","last_fullsized":"2023-10-14T01:27:11.447164+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28287541267,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2186043,"num_read_kb":8617764,"num_write":2181267,"num_write_kb":8725062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17866285056,"data_stored":27048648742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40276,"ondisk_log_size":40276,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48561,"ondisk_log_size":48561,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739363,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10177980,"kb_used_data":8722932,"kb_used_omap":1432,"kb_used_meta":1453607,"kb_avail":84193860,"statfs":{"total":96636764160,"available":86214512640,"internally_reserved":0,"allocated":8932282368,"data_stored":13522606898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1488494542},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.245,"5min":0.41699999999999998,"15min":0.36399999999999999},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.38800000000000001,"5min":8.8360000000000003,"15min":8.8360000000000003},"last":0.192},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.36499999999999999,"15min":0.437},"min":{"1min":0.215,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":0.82799999999999996,"5min":4.0069999999999997,"15min":7.9820000000000002},"last":0.223}]},{"osd":1,"last update":"Sat Oct 14 01:27:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":4.1929999999999996,"5min":11.952,"15min":15.845000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64700000000000002,"15min":0.57599999999999996},"min":{"1min":0.26300000000000001,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.2089999999999996,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.26900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739362,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5744704,"kb_used_data":4252600,"kb_used_omap":625,"kb_used_meta":1491470,"kb_avail":88627136,"statfs":{"total":96636764160,"available":90754187264,"internally_reserved":0,"allocated":4354662400,"data_stored":6595061686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1527266220},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.52800000000000002,"15min":0.63800000000000001},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.72299999999999998,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.57199999999999995,"15min":0.58799999999999997},"min":{"1min":0.27800000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.73999999999999999,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.374}]},{"osd":2,"last update":"Sat Oct 14 01:27:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.40400000000000003,"15min":0.46999999999999997},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.185},"max":{"1min":4.8319999999999999,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.48099999999999998,"15min":0.59099999999999997},"min":{"1min":0.23799999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65700000000000003,"5min":6.085,"15min":22.498999999999999},"last":0.22}]}]},{"osd":0,"up_from":8,"seq":34359739362,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6216172,"kb_used_data":4475296,"kb_used_omap":746,"kb_used_meta":1740117,"kb_avail":88155668,"statfs":{"total":96636764160,"available":90271404032,"internally_reserved":0,"allocated":4582703104,"data_stored":6931504979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1781880544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":0.58399999999999996,"15min":0.48499999999999999},"min":{"1min":0.193,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.023,"5min":16.023,"15min":16.023},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.17899999999999999},"max":{"1min":16.065000000000001,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16300000000000001},"max":{"1min":0.52800000000000002,"5min":4.2770000000000001,"15min":4.2770000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.316,"5min":0.373,"15min":0.442},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.44400000000000001,"5min":3.8220000000000001,"15min":4.8049999999999997},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4581347328,"data_stored":6931075072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4353130496,"data_stored":6594584595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8931807232,"data_stored":13522989075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:27:57.588 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:57.589 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:14.790 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:14.766+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:28:14.792 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:14.778+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:28:14.793 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:14.778+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:28:14.793 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:14.778+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:28:17.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:28:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:28:18.096 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2624,"stamp":"2023-10-14T01:28:17.619964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28325191397,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2229138,"num_read_kb":8683838,"num_write":2191490,"num_write_kb":8765019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88939,"ondisk_log_size":88939,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22412540,"kb_used_data":17530148,"kb_used_omap":2803,"kb_used_meta":4879564,"kb_avail":260702980,"statfs":{"total":289910292480,"available":266959851520,"internally_reserved":0,"allocated":17950871552,"data_stored":27130397243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4996674138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,4,6,15,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20512768,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5080,"num_read_kb":20264,"num_write":5226,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001907"},"pg_stats":[{"pgid":"2.7","version":"132'7293","reported_seq":10661,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.502969+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:28:17.502969+0000","last_peered":"2023-10-14T01:28:17.502969+0000","last_clean":"2023-10-14T01:28:17.502969+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:28:17.502969+0000","last_undegraded":"2023-10-14T01:28:17.502969+0000","last_fullsized":"2023-10-14T01:28:17.502969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'10863","reported_seq":17085,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.514847+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:28:17.514847+0000","last_peered":"2023-10-14T01:28:17.514847+0000","last_clean":"2023-10-14T01:28:17.514847+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:28:17.514847+0000","last_undegraded":"2023-10-14T01:28:17.514847+0000","last_fullsized":"2023-10-14T01:28:17.514847+0000","mapping_epoch":12,"log_start":"47'800","ondisk_log_start":"47'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"132'7271","reported_seq":10438,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.511246+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:28:17.511246+0000","last_peered":"2023-10-14T01:28:17.511246+0000","last_clean":"2023-10-14T01:28:17.511246+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:28:17.511246+0000","last_undegraded":"2023-10-14T01:28:17.511246+0000","last_fullsized":"2023-10-14T01:28:17.511246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7271,"log_dups_size":0,"ondisk_log_size":7271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'17925","reported_seq":26288,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.538407+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:28:17.538407+0000","last_peered":"2023-10-14T01:28:17.538407+0000","last_clean":"2023-10-14T01:28:17.538407+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:28:17.538407+0000","last_undegraded":"2023-10-14T01:28:17.538407+0000","last_fullsized":"2023-10-14T01:28:17.538407+0000","mapping_epoch":12,"log_start":"104'7900","ondisk_log_start":"104'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'570547","reported_seq":1135116,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:13.059136+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:28:13.059136+0000","last_peered":"2023-10-14T01:28:13.059136+0000","last_clean":"2023-10-14T01:28:13.059136+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:28:13.059136+0000","last_undegraded":"2023-10-14T01:28:13.059136+0000","last_fullsized":"2023-10-14T01:28:13.059136+0000","mapping_epoch":15,"log_start":"131'560536","ondisk_log_start":"131'560536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7316971520,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":566472,"num_read_kb":2233668,"num_write":565383,"num_write_kb":2261532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'3769","reported_seq":5541,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:11.128672+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:28:11.128672+0000","last_peered":"2023-10-14T01:28:11.128672+0000","last_clean":"2023-10-14T01:28:11.128672+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:28:11.128672+0000","last_undegraded":"2023-10-14T01:28:11.128672+0000","last_fullsized":"2023-10-14T01:28:11.128672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'541032","reported_seq":1077518,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:17.538814+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:28:17.538814+0000","last_peered":"2023-10-14T01:28:17.538814+0000","last_clean":"2023-10-14T01:28:17.538814+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:28:17.538814+0000","last_undegraded":"2023-10-14T01:28:17.538814+0000","last_fullsized":"2023-10-14T01:28:17.538814+0000","mapping_epoch":15,"log_start":"131'531024","ondisk_log_start":"131'531024","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1820,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 397s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6950957056,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":538254,"num_read_kb":2121856,"num_write":537374,"num_write_kb":2149496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3209,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:12.016893+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:28:12.016893+0000","last_peered":"2023-10-14T01:28:12.016893+0000","last_clean":"2023-10-14T01:28:12.016893+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:28:12.016893+0000","last_undegraded":"2023-10-14T01:28:12.016893+0000","last_fullsized":"2023-10-14T01:28:12.016893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'562508","reported_seq":1119489,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:13.066389+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:28:13.066389+0000","last_peered":"2023-10-14T01:28:13.066389+0000","last_clean":"2023-10-14T01:28:13.066389+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:28:13.066389+0000","last_undegraded":"2023-10-14T01:28:13.066389+0000","last_fullsized":"2023-10-14T01:28:13.066389+0000","mapping_epoch":15,"log_start":"131'552428","ondisk_log_start":"131'552428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7203348480,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":558835,"num_read_kb":2203360,"num_write":557446,"num_write_kb":2229784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3360,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:12.015411+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:28:12.015411+0000","last_peered":"2023-10-14T01:28:12.015411+0000","last_clean":"2023-10-14T01:28:12.015411+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:28:12.015411+0000","last_undegraded":"2023-10-14T01:28:12.015411+0000","last_fullsized":"2023-10-14T01:28:12.015411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'535315","reported_seq":1065370,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.544384+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:28:17.544384+0000","last_peered":"2023-10-14T01:28:17.544384+0000","last_clean":"2023-10-14T01:28:17.544384+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:28:17.544384+0000","last_undegraded":"2023-10-14T01:28:17.544384+0000","last_fullsized":"2023-10-14T01:28:17.544384+0000","mapping_epoch":15,"log_start":"131'525227","ondisk_log_start":"131'525227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6853320723,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":531829,"num_read_kb":2096148,"num_write":530491,"num_write_kb":2121958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"132'10808","reported_seq":15445,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.550402+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:28:17.550402+0000","last_peered":"2023-10-14T01:28:17.550402+0000","last_clean":"2023-10-14T01:28:17.550402+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:28:17.550402+0000","last_undegraded":"2023-10-14T01:28:17.550402+0000","last_fullsized":"2023-10-14T01:28:17.550402+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":311,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:11.447164+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:27:11.447164+0000","last_peered":"2023-10-14T01:27:11.447164+0000","last_clean":"2023-10-14T01:27:11.447164+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:27:11.447164+0000","last_undegraded":"2023-10-14T01:27:11.447164+0000","last_fullsized":"2023-10-14T01:27:11.447164+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28324597779,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2195390,"num_read_kb":8655032,"num_write":2190694,"num_write_kb":8762770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17947795456,"data_stored":27130159142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40187,"ondisk_log_size":40187,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48653,"ondisk_log_size":48653,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739368,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10318152,"kb_used_data":8764544,"kb_used_omap":1432,"kb_used_meta":1552167,"kb_avail":84053688,"statfs":{"total":96636764160,"available":86070976512,"internally_reserved":0,"allocated":8974893056,"data_stored":13565217586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1589419982},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:58 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.27500000000000002,"15min":0.36599999999999999},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":8.8360000000000003},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34200000000000003,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":1.052,"5min":1.532,"15min":7.9820000000000002},"last":0.29199999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:27:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":4.1929999999999996,"5min":11.952,"15min":15.845000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64700000000000002,"15min":0.57599999999999996},"min":{"1min":0.26300000000000001,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.2089999999999996,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.19500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739367,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5806888,"kb_used_data":4273120,"kb_used_omap":625,"kb_used_meta":1533134,"kb_avail":88564952,"statfs":{"total":96636764160,"available":90690510848,"internally_reserved":0,"allocated":4375674880,"data_stored":6616074166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1569930156},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.54500000000000004,"15min":0.63},"min":{"1min":0.193,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.484,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.59199999999999997,"15min":0.58099999999999996},"min":{"1min":0.28299999999999997,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":1.5069999999999999,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.39200000000000002,"15min":0.443},"min":{"1min":0.23899999999999999,"5min":0.193,"15min":0.185},"max":{"1min":0.40100000000000002,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.49199999999999999,"15min":0.58099999999999996},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.448,"5min":6.085,"15min":22.498999999999999},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739366,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6287500,"kb_used_data":4492484,"kb_used_omap":746,"kb_used_meta":1794261,"kb_avail":88084340,"statfs":{"total":96636764160,"available":90198364160,"internally_reserved":0,"allocated":4600303616,"data_stored":6949105491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1837324000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,8,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.56399999999999995,"15min":0.46700000000000003},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":0.48399999999999999,"5min":16.023,"15min":16.023},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.252,"5min":0.20799999999999999,"15min":0.17899999999999999},"max":{"1min":0.498,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16300000000000001},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.0609999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.72999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4599111680,"data_stored":6948839424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4373991424,"data_stored":6615445523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8974692352,"data_stored":13565874195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:28:18.098 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:18.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:28:18.100 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:28:18.098+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:28:18.566 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:28:18.594 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:34.188 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:34.186+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:28:34.198 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:34.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:28:34.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:34.202+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:28:34.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:34.202+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:28:38.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:39.027 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:28:39.026+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:28:39.129 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:28:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:28:39.157 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2635,"stamp":"2023-10-14T01:28:37.623366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28354072293,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2236538,"num_read_kb":8713338,"num_write":2198808,"num_write_kb":8794291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89081,"ondisk_log_size":89081,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22621132,"kb_used_data":17592564,"kb_used_omap":2803,"kb_used_meta":5025740,"kb_avail":260494388,"statfs":{"total":289910292480,"available":266746253312,"internally_reserved":0,"allocated":18014785536,"data_stored":27194299056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":5146358362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,7,4,15,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16617472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4237,"num_read_kb":16872,"num_write":4206,"num_write_kb":16824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001991"},"pg_stats":[{"pgid":"2.7","version":"133'7315","reported_seq":10690,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.499396+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:28:32.499396+0000","last_peered":"2023-10-14T01:28:32.499396+0000","last_clean":"2023-10-14T01:28:32.499396+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:28:32.499396+0000","last_undegraded":"2023-10-14T01:28:32.499396+0000","last_fullsized":"2023-10-14T01:28:32.499396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'10896","reported_seq":17128,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.499034+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:28:32.499034+0000","last_peered":"2023-10-14T01:28:32.499034+0000","last_clean":"2023-10-14T01:28:32.499034+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:28:32.499034+0000","last_undegraded":"2023-10-14T01:28:32.499034+0000","last_fullsized":"2023-10-14T01:28:32.499034+0000","mapping_epoch":12,"log_start":"47'800","ondisk_log_start":"47'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'7293","reported_seq":10467,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.498822+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:28:32.498822+0000","last_peered":"2023-10-14T01:28:32.498822+0000","last_clean":"2023-10-14T01:28:32.498822+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:28:32.498822+0000","last_undegraded":"2023-10-14T01:28:32.498822+0000","last_fullsized":"2023-10-14T01:28:32.498822+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'17980","reported_seq":26359,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.499584+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:28:32.499584+0000","last_peered":"2023-10-14T01:28:32.499584+0000","last_clean":"2023-10-14T01:28:32.499584+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:28:32.499584+0000","last_undegraded":"2023-10-14T01:28:32.499584+0000","last_fullsized":"2023-10-14T01:28:32.499584+0000","mapping_epoch":12,"log_start":"104'7900","ondisk_log_start":"104'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"133'572662","reported_seq":1139372,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:33.166343+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:28:33.166343+0000","last_peered":"2023-10-14T01:28:33.166343+0000","last_clean":"2023-10-14T01:28:33.166343+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:28:33.166343+0000","last_undegraded":"2023-10-14T01:28:33.166343+0000","last_fullsized":"2023-10-14T01:28:33.166343+0000","mapping_epoch":15,"log_start":"131'562636","ondisk_log_start":"131'562636","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7325294592,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":568613,"num_read_kb":2242204,"num_write":567498,"num_write_kb":2269992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"133'3785","reported_seq":5562,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:30.117848+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:28:30.117848+0000","last_peered":"2023-10-14T01:28:30.117848+0000","last_clean":"2023-10-14T01:28:30.117848+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:28:30.117848+0000","last_undegraded":"2023-10-14T01:28:30.117848+0000","last_fullsized":"2023-10-14T01:28:30.117848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3785,"log_dups_size":0,"ondisk_log_size":3785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"133'542570","reported_seq":1080552,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:32.665452+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:28:32.665452+0000","last_peered":"2023-10-14T01:28:32.665452+0000","last_clean":"2023-10-14T01:28:32.665452+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:28:32.665452+0000","last_undegraded":"2023-10-14T01:28:32.665452+0000","last_fullsized":"2023-10-14T01:28:32.665452+0000","mapping_epoch":15,"log_start":"131'532524","ondisk_log_start":"131'532524","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1882,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 413s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6956761088,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":539742,"num_read_kb":2127788,"num_write":538869,"num_write_kb":2155476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3218,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.495110+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:28:32.495110+0000","last_peered":"2023-10-14T01:28:32.495110+0000","last_clean":"2023-10-14T01:28:32.495110+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:28:32.495110+0000","last_undegraded":"2023-10-14T01:28:32.495110+0000","last_fullsized":"2023-10-14T01:28:32.495110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"133'564652","reported_seq":1123837,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:33.157195+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:28:33.157195+0000","last_peered":"2023-10-14T01:28:33.157195+0000","last_clean":"2023-10-14T01:28:33.157195+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:28:33.157195+0000","last_undegraded":"2023-10-14T01:28:33.157195+0000","last_fullsized":"2023-10-14T01:28:33.157195+0000","mapping_epoch":15,"log_start":"131'554628","ondisk_log_start":"131'554628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7211831296,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":561038,"num_read_kb":2212136,"num_write":559590,"num_write_kb":2238360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3369,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.509475+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:28:32.509475+0000","last_peered":"2023-10-14T01:28:32.509475+0000","last_clean":"2023-10-14T01:28:32.509475+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:28:32.509475+0000","last_undegraded":"2023-10-14T01:28:32.509475+0000","last_fullsized":"2023-10-14T01:28:32.509475+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"133'536879","reported_seq":1068496,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.667046+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:28:32.667046+0000","last_peered":"2023-10-14T01:28:32.667046+0000","last_clean":"2023-10-14T01:28:32.667046+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:28:32.667046+0000","last_undegraded":"2023-10-14T01:28:32.667046+0000","last_fullsized":"2023-10-14T01:28:32.667046+0000","mapping_epoch":15,"log_start":"131'526827","ondisk_log_start":"131'526827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6859591699,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":533397,"num_read_kb":2102404,"num_write":532055,"num_write_kb":2128214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"133'10841","reported_seq":15489,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.499178+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:28:32.499178+0000","last_peered":"2023-10-14T01:28:32.499178+0000","last_clean":"2023-10-14T01:28:32.499178+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:28:32.499178+0000","last_undegraded":"2023-10-14T01:28:32.499178+0000","last_fullsized":"2023-10-14T01:28:32.499178+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":312,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:42.330650+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:27:42.330650+0000","last_peered":"2023-10-14T01:27:42.330650+0000","last_clean":"2023-10-14T01:27:42.330650+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:27:42.330650+0000","last_undegraded":"2023-10-14T01:27:42.330650+0000","last_fullsized":"2023-10-14T01:27:42.330650+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28353478675,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2202790,"num_read_kb":8684532,"num_write":2198012,"num_write_kb":8792042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18012020736,"data_stored":27194384422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48834,"ondisk_log_size":48834,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10429264,"kb_used_data":8797768,"kb_used_omap":1432,"kb_used_meta":1630055,"kb_avail":83942576,"statfs":{"total":96636764160,"available":85957197824,"internally_reserved":0,"allocated":9008914432,"data_stored":13599234905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1669177294},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:58 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.27500000000000002,"15min":0.36599999999999999},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":8.8360000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34200000000000003,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":1.052,"5min":1.532,"15min":7.9820000000000002},"last":0.223}]},{"osd":1,"last update":"Sat Oct 14 01:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.60199999999999998,"15min":0.55400000000000005},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.8150000000000004,"5min":11.952,"15min":15.845000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.7729999999999997,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.40799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739370,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5849528,"kb_used_data":4285232,"kb_used_omap":625,"kb_used_meta":1563662,"kb_avail":88522312,"statfs":{"total":96636764160,"available":90646847488,"internally_reserved":0,"allocated":4388077568,"data_stored":6628472797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1601190828},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.54500000000000004,"15min":0.63},"min":{"1min":0.193,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.484,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.19},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.59199999999999997,"15min":0.58099999999999996},"min":{"1min":0.28299999999999997,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":1.5069999999999999,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.39200000000000002,"15min":0.443},"min":{"1min":0.23899999999999999,"5min":0.193,"15min":0.185},"max":{"1min":0.40100000000000002,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.49199999999999999,"15min":0.58099999999999996},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.448,"5min":6.085,"15min":22.498999999999999},"last":0.22700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739370,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6342340,"kb_used_data":4509564,"kb_used_omap":746,"kb_used_meta":1832021,"kb_avail":88029500,"statfs":{"total":96636764160,"available":90142208000,"internally_reserved":0,"allocated":4617793536,"data_stored":6966591354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1875990240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.56399999999999995,"15min":0.46700000000000003},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":0.48399999999999999,"5min":16.023,"15min":16.023},"last":0.318},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.252,"5min":0.20799999999999999,"15min":0.17899999999999999},"max":{"1min":0.498,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16300000000000001},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.0609999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4616564736,"data_stored":6966292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4386512896,"data_stored":6627966995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9008943104,"data_stored":13600124947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:28:39.159 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:39.159 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:56.016 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.013+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:28:56.028 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:28:56.028 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:28:56.029 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:28:56.029 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:28:56.029 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:28:56.030 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:28:56.030 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:28:56.030 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:28:56.030 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:28:56.031 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:28:56.031 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:28:56.025+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:28:59.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:28:59.748 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:28:59.781 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2646,"stamp":"2023-10-14T01:28:59.626911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28395060965,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2246731,"num_read_kb":8754002,"num_write":2209233,"num_write_kb":8835991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89198,"ondisk_log_size":89198,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22346872,"kb_used_data":17669724,"kb_used_omap":2803,"kb_used_meta":4674316,"kb_avail":260768648,"statfs":{"total":289910292480,"available":267027095552,"internally_reserved":0,"allocated":18093797376,"data_stored":27273310896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4786500186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,7,11,7,23,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24244224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5948,"num_read_kb":23720,"num_write":6152,"num_write_kb":24608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001917"},"pg_stats":[{"pgid":"2.7","version":"133'7357","reported_seq":10740,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.318226+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:28:57.318226+0000","last_peered":"2023-10-14T01:28:57.318226+0000","last_clean":"2023-10-14T01:28:57.318226+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:28:57.318226+0000","last_undegraded":"2023-10-14T01:28:57.318226+0000","last_fullsized":"2023-10-14T01:28:57.318226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'10960","reported_seq":17204,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.341828+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:28:57.341828+0000","last_peered":"2023-10-14T01:28:57.341828+0000","last_clean":"2023-10-14T01:28:57.341828+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:28:57.341828+0000","last_undegraded":"2023-10-14T01:28:57.341828+0000","last_fullsized":"2023-10-14T01:28:57.341828+0000","mapping_epoch":12,"log_start":"49'900","ondisk_log_start":"49'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'7336","reported_seq":10518,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.318091+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:28:57.318091+0000","last_peered":"2023-10-14T01:28:57.318091+0000","last_clean":"2023-10-14T01:28:57.318091+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:28:57.318091+0000","last_undegraded":"2023-10-14T01:28:57.318091+0000","last_fullsized":"2023-10-14T01:28:57.318091+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7336,"log_dups_size":0,"ondisk_log_size":7336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'18085","reported_seq":26484,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.341261+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:28:57.341261+0000","last_peered":"2023-10-14T01:28:57.341261+0000","last_clean":"2023-10-14T01:28:57.341261+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:28:57.341261+0000","last_undegraded":"2023-10-14T01:28:57.341261+0000","last_fullsized":"2023-10-14T01:28:57.341261+0000","mapping_epoch":12,"log_start":"106'8000","ondisk_log_start":"106'8000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"133'575420","reported_seq":1144712,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.334025+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:28:58.334025+0000","last_peered":"2023-10-14T01:28:58.334025+0000","last_clean":"2023-10-14T01:28:58.334025+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:28:58.334025+0000","last_undegraded":"2023-10-14T01:28:58.334025+0000","last_fullsized":"2023-10-14T01:28:58.334025+0000","mapping_epoch":15,"log_start":"132'565336","ondisk_log_start":"132'565336","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7336120320,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":571195,"num_read_kb":2252516,"num_write":570256,"num_write_kb":2281024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"133'3806","reported_seq":5588,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.754320+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:28:57.754320+0000","last_peered":"2023-10-14T01:28:57.754320+0000","last_clean":"2023-10-14T01:28:57.754320+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:28:57.754320+0000","last_undegraded":"2023-10-14T01:28:57.754320+0000","last_fullsized":"2023-10-14T01:28:57.754320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"133'545172","reported_seq":1085633,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:57.787925+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:28:57.787925+0000","last_peered":"2023-10-14T01:28:57.787925+0000","last_clean":"2023-10-14T01:28:57.787925+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:28:57.787925+0000","last_undegraded":"2023-10-14T01:28:57.787925+0000","last_fullsized":"2023-10-14T01:28:57.787925+0000","mapping_epoch":15,"log_start":"132'535124","ondisk_log_start":"132'535124","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":1966,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 438s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6967087104,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":542210,"num_read_kb":2137632,"num_write":541416,"num_write_kb":2165664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3227,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:53.019802+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:28:53.019802+0000","last_peered":"2023-10-14T01:28:53.019802+0000","last_clean":"2023-10-14T01:28:53.019802+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:28:53.019802+0000","last_undegraded":"2023-10-14T01:28:53.019802+0000","last_fullsized":"2023-10-14T01:28:53.019802+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"133'567312","reported_seq":1129159,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.314741+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:28:58.314741+0000","last_peered":"2023-10-14T01:28:58.314741+0000","last_clean":"2023-10-14T01:28:58.314741+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:28:58.314741+0000","last_undegraded":"2023-10-14T01:28:58.314741+0000","last_fullsized":"2023-10-14T01:28:58.314741+0000","mapping_epoch":15,"log_start":"132'557228","ondisk_log_start":"132'557228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7222095872,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":563699,"num_read_kb":2222748,"num_write":562250,"num_write_kb":2249000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3377,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:52.975695+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:28:52.975695+0000","last_peered":"2023-10-14T01:28:52.975695+0000","last_clean":"2023-10-14T01:28:52.975695+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:28:52.975695+0000","last_undegraded":"2023-10-14T01:28:52.975695+0000","last_fullsized":"2023-10-14T01:28:52.975695+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"133'539339","reported_seq":1073444,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.791271+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:28:57.791271+0000","last_peered":"2023-10-14T01:28:57.791271+0000","last_clean":"2023-10-14T01:28:57.791271+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:28:57.791271+0000","last_undegraded":"2023-10-14T01:28:57.791271+0000","last_fullsized":"2023-10-14T01:28:57.791271+0000","mapping_epoch":15,"log_start":"132'529327","ondisk_log_start":"132'529327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6869164051,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":535879,"num_read_kb":2112300,"num_write":534515,"num_write_kb":2138054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"133'10903","reported_seq":15563,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:57.325982+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:28:57.325982+0000","last_peered":"2023-10-14T01:28:57.325982+0000","last_clean":"2023-10-14T01:28:57.325982+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:28:57.325982+0000","last_undegraded":"2023-10-14T01:28:57.325982+0000","last_fullsized":"2023-10-14T01:28:57.325982+0000","mapping_epoch":12,"log_start":"49'900","ondisk_log_start":"49'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":312,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:42.330650+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:27:42.330650+0000","last_peered":"2023-10-14T01:27:42.330650+0000","last_clean":"2023-10-14T01:27:42.330650+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:27:42.330650+0000","last_undegraded":"2023-10-14T01:27:42.330650+0000","last_fullsized":"2023-10-14T01:27:42.330650+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28394467347,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2212983,"num_read_kb":8725196,"num_write":2208437,"num_write_kb":8833742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18088857600,"data_stored":27271221286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40228,"ondisk_log_size":40228,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48871,"ondisk_log_size":48871,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739376,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10600180,"kb_used_data":8832488,"kb_used_omap":1432,"kb_used_meta":1766247,"kb_avail":83771660,"statfs":{"total":96636764160,"available":85782179840,"internally_reserved":0,"allocated":9044467712,"data_stored":13634788185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1808637902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:58 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.27500000000000002,"15min":0.36599999999999999},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":8.8360000000000003},"last":4.8760000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34200000000000003,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":1.052,"5min":1.532,"15min":7.9820000000000002},"last":0.23000000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.60199999999999998,"15min":0.55400000000000005},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.8150000000000004,"5min":11.952,"15min":15.845000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.7729999999999997,"5min":12.029999999999999,"15min":15.805999999999999},"last":4.8419999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739375,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5911280,"kb_used_data":4305320,"kb_used_omap":625,"kb_used_meta":1605326,"kb_avail":88460560,"statfs":{"total":96636764160,"available":90583613440,"internally_reserved":0,"allocated":4408647680,"data_stored":6649042909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1643854764},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,1,18,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.54500000000000004,"15min":0.63},"min":{"1min":0.193,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.484,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.59199999999999997,"15min":0.58099999999999996},"min":{"1min":0.28299999999999997,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":1.5069999999999999,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.38400000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.39200000000000002,"15min":0.443},"min":{"1min":0.23899999999999999,"5min":0.193,"15min":0.185},"max":{"1min":0.40100000000000002,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":3.9580000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.49199999999999999,"15min":0.58099999999999996},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.448,"5min":6.085,"15min":22.498999999999999},"last":4.1379999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5835412,"kb_used_data":4531916,"kb_used_omap":746,"kb_used_meta":1302741,"kb_avail":88536428,"statfs":{"total":96636764160,"available":90661302272,"internally_reserved":0,"allocated":4640681984,"data_stored":6989479802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1334007520},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,5,6,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.56399999999999995,"15min":0.46700000000000003},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":0.48399999999999999,"5min":16.023,"15min":16.023},"last":0.308},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.252,"5min":0.20799999999999999,"15min":0.17899999999999999},"max":{"1min":0.498,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:28:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16300000000000001},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.0609999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4638756864,"data_stored":6988484608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4407009280,"data_stored":6648463379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9043091456,"data_stored":13634273299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:28:59.783 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:59.783 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:19.784 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:20.372 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:29:20.379 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:29:20.429 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2656,"stamp":"2023-10-14T01:29:19.634929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28428541669,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2255076,"num_read_kb":8787318,"num_write":2217783,"num_write_kb":8870191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89192,"ondisk_log_size":89192,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22609396,"kb_used_data":17737820,"kb_used_omap":2803,"kb_used_meta":4868748,"kb_avail":260506124,"statfs":{"total":289910292480,"available":266758270976,"internally_reserved":0,"allocated":18163527680,"data_stored":27343041200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":4985598554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,6,12,14,13,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25071616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6065,"num_read_kb":24216,"num_write":6411,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006635"},"pg_stats":[{"pgid":"2.7","version":"133'7365","reported_seq":10756,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.455499+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:29:13.455499+0000","last_peered":"2023-10-14T01:29:13.455499+0000","last_clean":"2023-10-14T01:29:13.455499+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:29:13.455499+0000","last_undegraded":"2023-10-14T01:29:13.455499+0000","last_fullsized":"2023-10-14T01:29:13.455499+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7365,"log_dups_size":0,"ondisk_log_size":7365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'10972","reported_seq":17228,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.455479+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:29:13.455479+0000","last_peered":"2023-10-14T01:29:13.455479+0000","last_clean":"2023-10-14T01:29:13.455479+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:29:13.455479+0000","last_undegraded":"2023-10-14T01:29:13.455479+0000","last_fullsized":"2023-10-14T01:29:13.455479+0000","mapping_epoch":12,"log_start":"49'900","ondisk_log_start":"49'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'7344","reported_seq":10535,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.455263+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:29:13.455263+0000","last_peered":"2023-10-14T01:29:13.455263+0000","last_clean":"2023-10-14T01:29:13.455263+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:29:13.455263+0000","last_undegraded":"2023-10-14T01:29:13.455263+0000","last_fullsized":"2023-10-14T01:29:13.455263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7344,"log_dups_size":0,"ondisk_log_size":7344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'18105","reported_seq":26525,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.455959+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:29:13.455959+0000","last_peered":"2023-10-14T01:29:13.455959+0000","last_clean":"2023-10-14T01:29:13.455959+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:29:13.455959+0000","last_undegraded":"2023-10-14T01:29:13.455959+0000","last_fullsized":"2023-10-14T01:29:13.455959+0000","mapping_epoch":12,"log_start":"106'8100","ondisk_log_start":"106'8100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"133'577596","reported_seq":1149049,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.415876+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:29:18.415876+0000","last_peered":"2023-10-14T01:29:18.415876+0000","last_clean":"2023-10-14T01:29:18.415876+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:29:18.415876+0000","last_undegraded":"2023-10-14T01:29:18.415876+0000","last_fullsized":"2023-10-14T01:29:18.415876+0000","mapping_epoch":15,"log_start":"132'567536","ondisk_log_start":"132'567536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7344795648,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":573357,"num_read_kb":2261148,"num_write":572432,"num_write_kb":2289728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"133'3819","reported_seq":5605,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:16.506846+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:29:16.506846+0000","last_peered":"2023-10-14T01:29:16.506846+0000","last_clean":"2023-10-14T01:29:16.506846+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:29:16.506846+0000","last_undegraded":"2023-10-14T01:29:16.506846+0000","last_fullsized":"2023-10-14T01:29:16.506846+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"133'547312","reported_seq":1089761,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:17.905950+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:29:17.905950+0000","last_peered":"2023-10-14T01:29:17.905950+0000","last_clean":"2023-10-14T01:29:17.905950+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:29:17.905950+0000","last_undegraded":"2023-10-14T01:29:17.905950+0000","last_fullsized":"2023-10-14T01:29:17.905950+0000","mapping_epoch":15,"log_start":"132'537225","ondisk_log_start":"132'537225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2069,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 458s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6975176704,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":544187,"num_read_kb":2145532,"num_write":543484,"num_write_kb":2173936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3235,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.647680+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:29:13.647680+0000","last_peered":"2023-10-14T01:29:13.647680+0000","last_clean":"2023-10-14T01:29:13.647680+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:29:13.647680+0000","last_undegraded":"2023-10-14T01:29:13.647680+0000","last_fullsized":"2023-10-14T01:29:13.647680+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"133'569514","reported_seq":1133515,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.412654+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:29:18.412654+0000","last_peered":"2023-10-14T01:29:18.412654+0000","last_clean":"2023-10-14T01:29:18.412654+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:29:18.412654+0000","last_undegraded":"2023-10-14T01:29:18.412654+0000","last_fullsized":"2023-10-14T01:29:18.412654+0000","mapping_epoch":15,"log_start":"132'559428","ondisk_log_start":"132'559428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7230672896,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":565853,"num_read_kb":2231340,"num_write":564452,"num_write_kb":2257808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3385,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.476611+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:29:13.476611+0000","last_peered":"2023-10-14T01:29:13.476611+0000","last_clean":"2023-10-14T01:29:13.476611+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:29:13.476611+0000","last_undegraded":"2023-10-14T01:29:13.476611+0000","last_fullsized":"2023-10-14T01:29:13.476611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"133'541443","reported_seq":1077602,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:17.893382+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:29:17.893382+0000","last_peered":"2023-10-14T01:29:17.893382+0000","last_clean":"2023-10-14T01:29:17.893382+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:29:17.893382+0000","last_undegraded":"2023-10-14T01:29:17.893382+0000","last_fullsized":"2023-10-14T01:29:17.893382+0000","mapping_epoch":15,"log_start":"132'531427","ondisk_log_start":"132'531427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6877302803,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":537931,"num_read_kb":2120492,"num_write":536619,"num_write_kb":2146470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"133'10915","reported_seq":15588,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:29:13.455676+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:29:13.455676+0000","last_peered":"2023-10-14T01:29:13.455676+0000","last_clean":"2023-10-14T01:29:13.455676+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:29:13.455676+0000","last_undegraded":"2023-10-14T01:29:13.455676+0000","last_fullsized":"2023-10-14T01:29:13.455676+0000","mapping_epoch":12,"log_start":"49'900","ondisk_log_start":"49'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":312,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-14T01:27:42.330650+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:27:42.330650+0000","last_peered":"2023-10-14T01:27:42.330650+0000","last_clean":"2023-10-14T01:27:42.330650+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:27:42.330650+0000","last_undegraded":"2023-10-14T01:27:42.330650+0000","last_fullsized":"2023-10-14T01:27:42.330650+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28427948051,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2221328,"num_read_kb":8758512,"num_write":2216987,"num_write_kb":8867942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18158747648,"data_stored":27341111334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40249,"ondisk_log_size":40249,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48844,"ondisk_log_size":48844,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10752204,"kb_used_data":8866436,"kb_used_omap":1432,"kb_used_meta":1884327,"kb_avail":83619636,"statfs":{"total":96636764160,"available":85626507264,"internally_reserved":0,"allocated":9079230464,"data_stored":13669550937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":1929551822},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.33000000000000002,"15min":0.38300000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.8760000000000003,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33200000000000002,"15min":0.40500000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":0.57299999999999995,"5min":1.532,"15min":7.9820000000000002},"last":0.27100000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.60199999999999998,"15min":0.55400000000000005},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.8150000000000004,"5min":11.952,"15min":15.845000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.7729999999999997,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739379,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5964056,"kb_used_data":4322000,"kb_used_omap":625,"kb_used_meta":1641422,"kb_avail":88407784,"statfs":{"total":96636764160,"available":90529570816,"internally_reserved":0,"allocated":4425728000,"data_stored":6666123229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1680817068},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,3,5,3,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51600000000000001,"15min":0.60899999999999999},"min":{"1min":0.189,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.79400000000000004,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":0.83499999999999996,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.45800000000000002},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.185},"max":{"1min":3.9580000000000002,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.57499999999999996},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.1379999999999999,"5min":6.085,"15min":22.498999999999999},"last":0.316}]}]},{"osd":0,"up_from":8,"seq":34359739379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5893136,"kb_used_data":4549384,"kb_used_omap":746,"kb_used_meta":1342997,"kb_avail":88478704,"statfs":{"total":96636764160,"available":90602192896,"internally_reserved":0,"allocated":4658569216,"data_stored":7007367034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1375229664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,7,6,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.51500000000000001,"15min":0.46700000000000003},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":1.446,"5min":16.023,"15min":16.023},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.54400000000000004,"15min":0.47099999999999997},"min":{"1min":0.23000000000000001,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":0.50900000000000001,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.308}]},{"osd":2,"last update":"Sat Oct 14 01:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51300000000000001,"15min":0.501},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.6120000000000001,"5min":5.141,"15min":5.141},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.46899999999999997,"15min":0.42999999999999999},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.72999999999999998,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":1.1819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4656676864,"data_stored":7006404608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4424097792,"data_stored":6665551891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9077972992,"data_stored":13669154835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:29:20.431 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:20.432 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:28.459 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:29:28.457+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:29:40.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:41.036 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:29:41.037 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:29:41.064 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2667,"stamp":"2023-10-14T01:29:39.638605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28460953317,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2263477,"num_read_kb":8820814,"num_write":2226056,"num_write_kb":8903283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89137,"ondisk_log_size":89137,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22902352,"kb_used_data":17803832,"kb_used_omap":2803,"kb_used_meta":5095692,"kb_avail":260213168,"statfs":{"total":289910292480,"available":266458284032,"internally_reserved":0,"allocated":18231123968,"data_stored":27410625188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":5217989210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,7,5,12,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16510976,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4253,"num_read_kb":16984,"num_write":4190,"num_write_kb":16760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001673"},"pg_stats":[{"pgid":"2.7","version":"134'7405","reported_seq":10804,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:36.623466+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:29:36.623466+0000","last_peered":"2023-10-14T01:29:36.623466+0000","last_clean":"2023-10-14T01:29:36.623466+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:29:36.623466+0000","last_undegraded":"2023-10-14T01:29:36.623466+0000","last_fullsized":"2023-10-14T01:29:36.623466+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'11032","reported_seq":17301,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:36.625725+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:29:36.625725+0000","last_peered":"2023-10-14T01:29:36.625725+0000","last_clean":"2023-10-14T01:29:36.625725+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:29:36.625725+0000","last_undegraded":"2023-10-14T01:29:36.625725+0000","last_fullsized":"2023-10-14T01:29:36.625725+0000","mapping_epoch":12,"log_start":"49'1000","ondisk_log_start":"49'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'7384","reported_seq":10583,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:36.611875+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:29:36.611875+0000","last_peered":"2023-10-14T01:29:36.611875+0000","last_clean":"2023-10-14T01:29:36.611875+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:29:36.611875+0000","last_undegraded":"2023-10-14T01:29:36.611875+0000","last_fullsized":"2023-10-14T01:29:36.611875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'18205","reported_seq":26644,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:36.637532+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:29:36.637532+0000","last_peered":"2023-10-14T01:29:36.637532+0000","last_clean":"2023-10-14T01:29:36.637532+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:29:36.637532+0000","last_undegraded":"2023-10-14T01:29:36.637532+0000","last_fullsized":"2023-10-14T01:29:36.637532+0000","mapping_epoch":12,"log_start":"108'8200","ondisk_log_start":"108'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'579747","reported_seq":1153396,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.488619+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:29:38.488619+0000","last_peered":"2023-10-14T01:29:38.488619+0000","last_clean":"2023-10-14T01:29:38.488619+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:29:38.488619+0000","last_undegraded":"2023-10-14T01:29:38.488619+0000","last_fullsized":"2023-10-14T01:29:38.488619+0000","mapping_epoch":15,"log_start":"132'569736","ondisk_log_start":"132'569736","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7353221120,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":575550,"num_read_kb":2269892,"num_write":574583,"num_write_kb":2298332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'3828","reported_seq":5619,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:34.849411+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:29:34.849411+0000","last_peered":"2023-10-14T01:29:34.849411+0000","last_clean":"2023-10-14T01:29:34.849411+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:29:34.849411+0000","last_undegraded":"2023-10-14T01:29:34.849411+0000","last_fullsized":"2023-10-14T01:29:34.849411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3828,"log_dups_size":0,"ondisk_log_size":3828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'549360","reported_seq":1093935,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:38.015316+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:29:38.015316+0000","last_peered":"2023-10-14T01:29:38.015316+0000","last_clean":"2023-10-14T01:29:38.015316+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:29:38.015316+0000","last_undegraded":"2023-10-14T01:29:38.015316+0000","last_fullsized":"2023-10-14T01:29:38.015316+0000","mapping_epoch":15,"log_start":"132'539325","ondisk_log_start":"132'539325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2159,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 478s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6982930432,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":546303,"num_read_kb":2153976,"num_write":545469,"num_write_kb":2181876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3244,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:33.937561+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:29:33.937561+0000","last_peered":"2023-10-14T01:29:33.937561+0000","last_clean":"2023-10-14T01:29:33.937561+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:29:33.937561+0000","last_undegraded":"2023-10-14T01:29:33.937561+0000","last_fullsized":"2023-10-14T01:29:33.937561+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'571551","reported_seq":1137682,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.489524+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:29:38.489524+0000","last_peered":"2023-10-14T01:29:38.489524+0000","last_clean":"2023-10-14T01:29:38.489524+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:29:38.489524+0000","last_undegraded":"2023-10-14T01:29:38.489524+0000","last_fullsized":"2023-10-14T01:29:38.489524+0000","mapping_epoch":15,"log_start":"132'561528","ondisk_log_start":"132'561528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7238590464,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":567985,"num_read_kb":2239840,"num_write":566489,"num_write_kb":2265956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3394,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:33.948221+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:29:33.948221+0000","last_peered":"2023-10-14T01:29:33.948221+0000","last_clean":"2023-10-14T01:29:33.948221+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:29:33.948221+0000","last_undegraded":"2023-10-14T01:29:33.948221+0000","last_fullsized":"2023-10-14T01:29:33.948221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'543543","reported_seq":1081661,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.030029+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:29:38.030029+0000","last_peered":"2023-10-14T01:29:38.030029+0000","last_clean":"2023-10-14T01:29:38.030029+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:29:38.030029+0000","last_undegraded":"2023-10-14T01:29:38.030029+0000","last_fullsized":"2023-10-14T01:29:38.030029+0000","mapping_epoch":15,"log_start":"132'533527","ondisk_log_start":"132'533527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6885617683,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":539891,"num_read_kb":2128300,"num_write":538719,"num_write_kb":2154870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'10975","reported_seq":15659,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:36.652988+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:29:36.652988+0000","last_peered":"2023-10-14T01:29:36.652988+0000","last_clean":"2023-10-14T01:29:36.652988+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:29:36.652988+0000","last_undegraded":"2023-10-14T01:29:36.652988+0000","last_fullsized":"2023-10-14T01:29:36.652988+0000","mapping_epoch":12,"log_start":"49'900","ondisk_log_start":"49'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":313,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:28:21.471747+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:28:21.471747+0000","last_peered":"2023-10-14T01:28:21.471747+0000","last_clean":"2023-10-14T01:28:21.471747+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:28:21.471747+0000","last_undegraded":"2023-10-14T01:28:21.471747+0000","last_fullsized":"2023-10-14T01:28:21.471747+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28460359699,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2229729,"num_read_kb":8792008,"num_write":2225260,"num_write_kb":8901034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18226511872,"data_stored":27408875558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40085,"ondisk_log_size":40085,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":48953,"ondisk_log_size":48953,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739384,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10875316,"kb_used_data":8899372,"kb_used_omap":1432,"kb_used_meta":1974503,"kb_avail":83496524,"statfs":{"total":96636764160,"available":85500440576,"internally_reserved":0,"allocated":9112956928,"data_stored":13703273301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":2021892046},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.33000000000000002,"15min":0.38300000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.8760000000000003,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":3.3319999999999999},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33200000000000002,"15min":0.40500000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":0.57299999999999995,"5min":1.532,"15min":7.9820000000000002},"last":3.1749999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:29:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.58599999999999997,"15min":0.51800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51600000000000001,"5min":11.952,"15min":15.845000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.59599999999999997},"min":{"1min":0.20999999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.8419999999999996,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.49099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739383,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6063052,"kb_used_data":4338372,"kb_used_omap":625,"kb_used_meta":1724046,"kb_avail":88308788,"statfs":{"total":96636764160,"available":90428198912,"internally_reserved":0,"allocated":4442492928,"data_stored":6682884057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1765424044},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,1,7,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51600000000000001,"15min":0.60899999999999999},"min":{"1min":0.189,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.79400000000000004,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":0.83499999999999996,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.45800000000000002},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.185},"max":{"1min":3.9580000000000002,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.57499999999999996},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.1379999999999999,"5min":6.085,"15min":22.498999999999999},"last":1.2869999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5963984,"kb_used_data":4566088,"kb_used_omap":746,"kb_used_meta":1397141,"kb_avail":88407856,"statfs":{"total":96636764160,"available":90529644544,"internally_reserved":0,"allocated":4675674112,"data_stored":7024467830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1430673120},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.51500000000000001,"15min":0.46700000000000003},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":1.446,"5min":16.023,"15min":16.023},"last":0.255},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.54400000000000004,"15min":0.47099999999999997},"min":{"1min":0.23000000000000001,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":0.50900000000000001,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51300000000000001,"15min":0.501},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.6120000000000001,"5min":5.141,"15min":5.141},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.46899999999999997,"15min":0.42999999999999999},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.72999999999999998,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4673835008,"data_stored":7023562752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4440834048,"data_stored":6682288147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9111842816,"data_stored":13703024659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:29:41.066 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:41.067 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:50.399 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:29:50.396+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:29:50.399 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:29:50.396+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:29:57.819 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:29:57.800+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:30:00.621 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:00.616+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:30:00.629 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:00.624+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:30:00.629 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:00.624+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:30:00.629 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:00.624+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:30:00.630 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:00.624+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:30:01.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:01.557 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:30:01.557 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:30:01.582 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2677,"stamp":"2023-10-14T01:29:59.642535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28493889253,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2271726,"num_read_kb":8853694,"num_write":2234484,"num_write_kb":8936995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89330,"ondisk_log_size":89330,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23143332,"kb_used_data":17871116,"kb_used_omap":2803,"kb_used_meta":5269388,"kb_avail":259972188,"statfs":{"total":289910292480,"available":266211520512,"internally_reserved":0,"allocated":18300022784,"data_stored":27479524004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":5395853914},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,4,10,17,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24702976,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6327,"num_read_kb":25204,"num_write":6306,"num_write_kb":25224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001865"},"pg_stats":[{"pgid":"2.7","version":"134'7432","reported_seq":10839,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.036877+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:29:58.036877+0000","last_peered":"2023-10-14T01:29:58.036877+0000","last_clean":"2023-10-14T01:29:58.036877+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:29:58.036877+0000","last_undegraded":"2023-10-14T01:29:58.036877+0000","last_fullsized":"2023-10-14T01:29:58.036877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'11072","reported_seq":17353,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.044204+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:29:58.044204+0000","last_peered":"2023-10-14T01:29:58.044204+0000","last_clean":"2023-10-14T01:29:58.044204+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:29:58.044204+0000","last_undegraded":"2023-10-14T01:29:58.044204+0000","last_fullsized":"2023-10-14T01:29:58.044204+0000","mapping_epoch":12,"log_start":"49'1000","ondisk_log_start":"49'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'7411","reported_seq":10618,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.044011+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:29:58.044011+0000","last_peered":"2023-10-14T01:29:58.044011+0000","last_clean":"2023-10-14T01:29:58.044011+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:29:58.044011+0000","last_undegraded":"2023-10-14T01:29:58.044011+0000","last_fullsized":"2023-10-14T01:29:58.044011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'18270","reported_seq":26729,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.058700+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:29:58.058700+0000","last_peered":"2023-10-14T01:29:58.058700+0000","last_clean":"2023-10-14T01:29:58.058700+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:29:58.058700+0000","last_undegraded":"2023-10-14T01:29:58.058700+0000","last_fullsized":"2023-10-14T01:29:58.058700+0000","mapping_epoch":12,"log_start":"108'8200","ondisk_log_start":"108'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'581962","reported_seq":1157741,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.601025+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:29:58.601025+0000","last_peered":"2023-10-14T01:29:58.601025+0000","last_clean":"2023-10-14T01:29:58.601025+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:29:58.601025+0000","last_undegraded":"2023-10-14T01:29:58.601025+0000","last_fullsized":"2023-10-14T01:29:58.601025+0000","mapping_epoch":15,"log_start":"133'571936","ondisk_log_start":"133'571936","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7362052096,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":577677,"num_read_kb":2278372,"num_write":576798,"num_write_kb":2307192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'3852","reported_seq":5647,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:57.769296+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:29:57.769296+0000","last_peered":"2023-10-14T01:29:57.769296+0000","last_clean":"2023-10-14T01:29:57.769296+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:29:57.769296+0000","last_undegraded":"2023-10-14T01:29:57.769296+0000","last_fullsized":"2023-10-14T01:29:57.769296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3852,"log_dups_size":0,"ondisk_log_size":3852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'551498","reported_seq":1098022,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:58.156875+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:29:58.156875+0000","last_peered":"2023-10-14T01:29:58.156875+0000","last_clean":"2023-10-14T01:29:58.156875+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:29:58.156875+0000","last_undegraded":"2023-10-14T01:29:58.156875+0000","last_fullsized":"2023-10-14T01:29:58.156875+0000","mapping_epoch":15,"log_start":"133'541425","ondisk_log_start":"133'541425","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2222,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 498s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6991073280,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":548243,"num_read_kb":2161712,"num_write":547564,"num_write_kb":2190256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3252,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:54.419279+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:29:54.419279+0000","last_peered":"2023-10-14T01:29:54.419279+0000","last_clean":"2023-10-14T01:29:54.419279+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:29:54.419279+0000","last_undegraded":"2023-10-14T01:29:54.419279+0000","last_fullsized":"2023-10-14T01:29:54.419279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'573673","reported_seq":1141950,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.598948+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:29:58.598948+0000","last_peered":"2023-10-14T01:29:58.598948+0000","last_clean":"2023-10-14T01:29:58.598948+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:29:58.598948+0000","last_undegraded":"2023-10-14T01:29:58.598948+0000","last_fullsized":"2023-10-14T01:29:58.598948+0000","mapping_epoch":15,"log_start":"133'563628","ondisk_log_start":"133'563628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7246712832,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":570128,"num_read_kb":2248384,"num_write":568611,"num_write_kb":2274444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3403,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:54.420083+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:29:54.420083+0000","last_peered":"2023-10-14T01:29:54.420083+0000","last_clean":"2023-10-14T01:29:54.420083+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:29:54.420083+0000","last_undegraded":"2023-10-14T01:29:54.420083+0000","last_fullsized":"2023-10-14T01:29:54.420083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'545539","reported_seq":1085699,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.163516+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:29:58.163516+0000","last_peered":"2023-10-14T01:29:58.163516+0000","last_clean":"2023-10-14T01:29:58.163516+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:29:58.163516+0000","last_undegraded":"2023-10-14T01:29:58.163516+0000","last_fullsized":"2023-10-14T01:29:58.163516+0000","mapping_epoch":15,"log_start":"132'535527","ondisk_log_start":"132'535527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6893457427,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":541930,"num_read_kb":2136420,"num_write":540715,"num_write_kb":2162854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'11014","reported_seq":15710,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.087980+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:29:58.087980+0000","last_peered":"2023-10-14T01:29:58.087980+0000","last_clean":"2023-10-14T01:29:58.087980+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:29:58.087980+0000","last_undegraded":"2023-10-14T01:29:58.087980+0000","last_fullsized":"2023-10-14T01:29:58.087980+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":314,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:31.492328+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:29:31.492328+0000","last_peered":"2023-10-14T01:29:31.492328+0000","last_clean":"2023-10-14T01:29:31.492328+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:29:31.492328+0000","last_undegraded":"2023-10-14T01:29:31.492328+0000","last_fullsized":"2023-10-14T01:29:31.492328+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28493295635,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2237978,"num_read_kb":8824888,"num_write":2233688,"num_write_kb":8934746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18295521280,"data_stored":27477884966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40156,"ondisk_log_size":40156,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49075,"ondisk_log_size":49075,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739388,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11031252,"kb_used_data":8933196,"kb_used_omap":1432,"kb_used_meta":2096615,"kb_avail":83340588,"statfs":{"total":96636764160,"available":85340762112,"internally_reserved":0,"allocated":9147592704,"data_stored":13737909077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":2146934734},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.33000000000000002,"15min":0.38300000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.8760000000000003,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.752},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33200000000000002,"15min":0.40500000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":0.57299999999999995,"5min":1.532,"15min":7.9820000000000002},"last":0.63700000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:29:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.58599999999999997,"15min":0.51800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51600000000000001,"5min":11.952,"15min":15.845000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.59599999999999997},"min":{"1min":0.20999999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.8419999999999996,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.40500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739387,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6115336,"kb_used_data":4354560,"kb_used_omap":625,"kb_used_meta":1760142,"kb_avail":88256504,"statfs":{"total":96636764160,"available":90374660096,"internally_reserved":0,"allocated":4459069440,"data_stored":6699460569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1802386348},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.51600000000000001,"15min":0.60899999999999999},"min":{"1min":0.189,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.79400000000000004,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":0.83499999999999996,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.64800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.45800000000000002},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.185},"max":{"1min":3.9580000000000002,"5min":4.8319999999999999,"15min":5.4790000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.57499999999999996},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.1379999999999999,"5min":6.085,"15min":22.498999999999999},"last":0.247}]}]},{"osd":0,"up_from":8,"seq":34359739387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5996744,"kb_used_data":4583360,"kb_used_omap":746,"kb_used_meta":1412629,"kb_avail":88375096,"statfs":{"total":96636764160,"available":90496098304,"internally_reserved":0,"allocated":4693360640,"data_stored":7042154358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1446532832},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.51500000000000001,"15min":0.46700000000000003},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":1.446,"5min":16.023,"15min":16.023},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.54400000000000004,"15min":0.47099999999999997},"min":{"1min":0.23000000000000001,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":0.50900000000000001,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51300000000000001,"15min":0.501},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.6120000000000001,"5min":5.141,"15min":5.141},"last":2.4350000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.46899999999999997,"15min":0.42999999999999999},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.72999999999999998,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4691599360,"data_stored":7041327104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4457594880,"data_stored":6699048979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9146327040,"data_stored":13737508883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:30:01.584 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:30:01.584 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:03.249 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:30:03.249 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:30:03.249 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:30:03.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:30:03.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:30:03.250 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:30:03.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:30:03.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:30:03.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:30:03.251 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:30:03.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:30:03.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:30:03.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:30:03.252 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:30:03.253 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:03.244+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:30:13.695 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:30:13.692+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:30:13.696 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:30:13.692+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:30:20.983 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:20.972+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:30:20.984 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:20.976+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:30:20.984 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:20.976+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:30:20.990 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:20.984+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:30:20.992 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:20.984+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:30:21.586 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:22.011 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:30:22.011 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:30:22.038 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2688,"stamp":"2023-10-14T01:30:21.648245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28526235365,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2280052,"num_read_kb":8886874,"num_write":2242735,"num_write_kb":8969999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89415,"ondisk_log_size":89415,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23337340,"kb_used_data":17935524,"kb_used_omap":2803,"kb_used_meta":5398988,"kb_avail":259778180,"statfs":{"total":289910292480,"available":266012856320,"internally_reserved":0,"allocated":18365976576,"data_stored":27545477796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":5528564314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,8,12,12,15,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16650240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4216,"num_read_kb":16784,"num_write":4270,"num_write_kb":17080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"2.7","version":"134'7468","reported_seq":10883,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:15.902267+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:30:15.902267+0000","last_peered":"2023-10-14T01:30:15.902267+0000","last_clean":"2023-10-14T01:30:15.902267+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:30:15.902267+0000","last_undegraded":"2023-10-14T01:30:15.902267+0000","last_fullsized":"2023-10-14T01:30:15.902267+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'11125","reported_seq":17418,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:15.907826+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:30:15.907826+0000","last_peered":"2023-10-14T01:30:15.907826+0000","last_clean":"2023-10-14T01:30:15.907826+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:30:15.907826+0000","last_undegraded":"2023-10-14T01:30:15.907826+0000","last_fullsized":"2023-10-14T01:30:15.907826+0000","mapping_epoch":12,"log_start":"51'1100","ondisk_log_start":"51'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'7447","reported_seq":10661,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:15.900835+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:30:15.900835+0000","last_peered":"2023-10-14T01:30:15.900835+0000","last_clean":"2023-10-14T01:30:15.900835+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:30:15.900835+0000","last_undegraded":"2023-10-14T01:30:15.900835+0000","last_fullsized":"2023-10-14T01:30:15.900835+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7447,"log_dups_size":0,"ondisk_log_size":7447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'18357","reported_seq":26834,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:15.920977+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:30:15.920977+0000","last_peered":"2023-10-14T01:30:15.920977+0000","last_clean":"2023-10-14T01:30:15.920977+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:30:15.920977+0000","last_undegraded":"2023-10-14T01:30:15.920977+0000","last_fullsized":"2023-10-14T01:30:15.920977+0000","mapping_epoch":12,"log_start":"108'8300","ondisk_log_start":"108'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'584066","reported_seq":1161983,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:18.705699+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:30:18.705699+0000","last_peered":"2023-10-14T01:30:18.705699+0000","last_clean":"2023-10-14T01:30:18.705699+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:30:18.705699+0000","last_undegraded":"2023-10-14T01:30:18.705699+0000","last_fullsized":"2023-10-14T01:30:18.705699+0000","mapping_epoch":15,"log_start":"133'574036","ondisk_log_start":"133'574036","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7370256384,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":579818,"num_read_kb":2286908,"num_write":578902,"num_write_kb":2315608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'3866","reported_seq":5665,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:14.894946+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:30:14.894946+0000","last_peered":"2023-10-14T01:30:14.894946+0000","last_clean":"2023-10-14T01:30:14.894946+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:30:14.894946+0000","last_undegraded":"2023-10-14T01:30:14.894946+0000","last_fullsized":"2023-10-14T01:30:14.894946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'553564","reported_seq":1102177,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:30:18.332705+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:30:18.332705+0000","last_peered":"2023-10-14T01:30:18.332705+0000","last_clean":"2023-10-14T01:30:18.332705+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:30:18.332705+0000","last_undegraded":"2023-10-14T01:30:18.332705+0000","last_fullsized":"2023-10-14T01:30:18.332705+0000","mapping_epoch":15,"log_start":"133'543525","ondisk_log_start":"133'543525","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2304,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 518s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":6999052288,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":550320,"num_read_kb":2169992,"num_write":549574,"num_write_kb":2198296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3260,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:14.895037+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:30:14.895037+0000","last_peered":"2023-10-14T01:30:14.895037+0000","last_clean":"2023-10-14T01:30:14.895037+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:30:14.895037+0000","last_undegraded":"2023-10-14T01:30:14.895037+0000","last_fullsized":"2023-10-14T01:30:14.895037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'575816","reported_seq":1146182,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:18.701570+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:30:18.701570+0000","last_peered":"2023-10-14T01:30:18.701570+0000","last_clean":"2023-10-14T01:30:18.701570+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:30:18.701570+0000","last_undegraded":"2023-10-14T01:30:18.701570+0000","last_fullsized":"2023-10-14T01:30:18.701570+0000","mapping_epoch":15,"log_start":"133'565728","ondisk_log_start":"133'565728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7255003136,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":572217,"num_read_kb":2256708,"num_write":570754,"num_write_kb":2283016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3411,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:14.900225+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:30:14.900225+0000","last_peered":"2023-10-14T01:30:14.900225+0000","last_clean":"2023-10-14T01:30:14.900225+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:30:14.900225+0000","last_undegraded":"2023-10-14T01:30:14.900225+0000","last_fullsized":"2023-10-14T01:30:14.900225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'547533","reported_seq":1089712,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:18.333526+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:30:18.333526+0000","last_peered":"2023-10-14T01:30:18.333526+0000","last_clean":"2023-10-14T01:30:18.333526+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:30:18.333526+0000","last_undegraded":"2023-10-14T01:30:18.333526+0000","last_fullsized":"2023-10-14T01:30:18.333526+0000","mapping_epoch":15,"log_start":"133'537527","ondisk_log_start":"133'537527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6901329939,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":543949,"num_read_kb":2144460,"num_write":542709,"num_write_kb":2170830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"134'11066","reported_seq":15772,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:15.931217+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:30:15.931217+0000","last_peered":"2023-10-14T01:30:15.931217+0000","last_clean":"2023-10-14T01:30:15.931217+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:30:15.931217+0000","last_undegraded":"2023-10-14T01:30:15.931217+0000","last_fullsized":"2023-10-14T01:30:15.931217+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":314,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:31.492328+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:29:31.492328+0000","last_peered":"2023-10-14T01:29:31.492328+0000","last_clean":"2023-10-14T01:29:31.492328+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:29:31.492328+0000","last_undegraded":"2023-10-14T01:29:31.492328+0000","last_fullsized":"2023-10-14T01:29:31.492328+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28525641747,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2246304,"num_read_kb":8858068,"num_write":2241939,"num_write_kb":8967750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18362949632,"data_stored":27545313318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40163,"ondisk_log_size":40163,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49153,"ondisk_log_size":49153,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739392,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11079916,"kb_used_data":8964708,"kb_used_omap":1432,"kb_used_meta":2113767,"kb_avail":83291924,"statfs":{"total":96636764160,"available":85290930176,"internally_reserved":0,"allocated":9179860992,"data_stored":13770177365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":2164498382},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.38400000000000001,"15min":0.39500000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.3319999999999999,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.40400000000000003,"15min":0.41299999999999998},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.752}]},{"osd":1,"last update":"Sat Oct 14 01:29:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.58599999999999997,"15min":0.51800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51600000000000001,"5min":11.952,"15min":15.845000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.59599999999999997},"min":{"1min":0.20999999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.8419999999999996,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.56999999999999995}]}]},{"osd":1,"up_from":8,"seq":34359739391,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6185568,"kb_used_data":4370648,"kb_used_omap":625,"kb_used_meta":1814286,"kb_avail":88186272,"statfs":{"total":96636764160,"available":90302742528,"internally_reserved":0,"allocated":4475543552,"data_stored":6715934681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1857829804},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,2,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51100000000000001,"15min":0.60799999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.399,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":3.488,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.378}]},{"osd":2,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.46600000000000003,"15min":0.47899999999999998},"min":{"1min":0.214,"5min":0.193,"15min":0.185},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.316}]}]},{"osd":0,"up_from":8,"seq":34359739391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6071856,"kb_used_data":4600168,"kb_used_omap":746,"kb_used_meta":1470933,"kb_avail":88299984,"statfs":{"total":96636764160,"available":90419183616,"internally_reserved":0,"allocated":4710572032,"data_stored":7059365750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1506236128},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,6,9,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.63800000000000001,"15min":0.497},"min":{"1min":0.17699999999999999,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":5.2359999999999998,"5min":16.023,"15min":16.023},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.48799999999999999},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":5.1909999999999998,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46100000000000002,"15min":0.499},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.4350000000000001,"5min":5.141,"15min":5.141},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.22700000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.1819999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4708990976,"data_stored":7058718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4473991168,"data_stored":6715445267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9179967488,"data_stored":13771149331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:30:22.040 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:30:22.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:30:22.506 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:30:22.540 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:30.358 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:30.348+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:30:42.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:43.041 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:30:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:30:43.098 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2699,"stamp":"2023-10-14T01:30:41.651720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28557745893,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2288118,"num_read_kb":8919050,"num_write":2250729,"num_write_kb":9001975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89516,"ondisk_log_size":89516,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23370128,"kb_used_data":17999472,"kb_used_omap":2803,"kb_used_meta":5367820,"kb_avail":259745392,"statfs":{"total":289910292480,"available":265979281408,"internally_reserved":0,"allocated":18431459328,"data_stored":27610952262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":5496648282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,9,4,11,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15503360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3999,"num_read_kb":15944,"num_write":3960,"num_write_kb":15840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001778"},"pg_stats":[{"pgid":"2.7","version":"135'7499","reported_seq":10922,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:37.822659+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:30:37.822659+0000","last_peered":"2023-10-14T01:30:37.822659+0000","last_clean":"2023-10-14T01:30:37.822659+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:30:37.822659+0000","last_undegraded":"2023-10-14T01:30:37.822659+0000","last_fullsized":"2023-10-14T01:30:37.822659+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'11172","reported_seq":17477,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:37.836549+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:30:37.836549+0000","last_peered":"2023-10-14T01:30:37.836549+0000","last_clean":"2023-10-14T01:30:37.836549+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:30:37.836549+0000","last_undegraded":"2023-10-14T01:30:37.836549+0000","last_fullsized":"2023-10-14T01:30:37.836549+0000","mapping_epoch":12,"log_start":"51'1100","ondisk_log_start":"51'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'7479","reported_seq":10701,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:37.819984+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:30:37.819984+0000","last_peered":"2023-10-14T01:30:37.819984+0000","last_clean":"2023-10-14T01:30:37.819984+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:30:37.819984+0000","last_undegraded":"2023-10-14T01:30:37.819984+0000","last_fullsized":"2023-10-14T01:30:37.819984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'18433","reported_seq":26929,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:37.843543+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:30:37.843543+0000","last_peered":"2023-10-14T01:30:37.843543+0000","last_clean":"2023-10-14T01:30:37.843543+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:30:37.843543+0000","last_undegraded":"2023-10-14T01:30:37.843543+0000","last_fullsized":"2023-10-14T01:30:37.843543+0000","mapping_epoch":12,"log_start":"108'8400","ondisk_log_start":"108'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'586014","reported_seq":1165968,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:38.755448+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:30:38.755448+0000","last_peered":"2023-10-14T01:30:38.755448+0000","last_clean":"2023-10-14T01:30:38.755448+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:30:38.755448+0000","last_undegraded":"2023-10-14T01:30:38.755448+0000","last_fullsized":"2023-10-14T01:30:38.755448+0000","mapping_epoch":15,"log_start":"133'575936","ondisk_log_start":"133'575936","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7377866752,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":581852,"num_read_kb":2295020,"num_write":580850,"num_write_kb":2323400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"134'3883","reported_seq":5687,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:35.990653+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:30:35.990653+0000","last_peered":"2023-10-14T01:30:35.990653+0000","last_clean":"2023-10-14T01:30:35.990653+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:30:35.990653+0000","last_undegraded":"2023-10-14T01:30:35.990653+0000","last_fullsized":"2023-10-14T01:30:35.990653+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'555615","reported_seq":1106260,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:30:38.431241+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:30:38.431241+0000","last_peered":"2023-10-14T01:30:38.431241+0000","last_clean":"2023-10-14T01:30:38.431241+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:30:38.431241+0000","last_undegraded":"2023-10-14T01:30:38.431241+0000","last_fullsized":"2023-10-14T01:30:38.431241+0000","mapping_epoch":15,"log_start":"133'545526","ondisk_log_start":"133'545526","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2393,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 538s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7006969856,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":552339,"num_read_kb":2178048,"num_write":551567,"num_write_kb":2206268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3269,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:35.375640+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:30:35.375640+0000","last_peered":"2023-10-14T01:30:35.375640+0000","last_clean":"2023-10-14T01:30:35.375640+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:30:35.375640+0000","last_undegraded":"2023-10-14T01:30:35.375640+0000","last_fullsized":"2023-10-14T01:30:35.375640+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'577846","reported_seq":1150318,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:38.754640+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:30:38.754640+0000","last_peered":"2023-10-14T01:30:38.754640+0000","last_clean":"2023-10-14T01:30:38.754640+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:30:38.754640+0000","last_undegraded":"2023-10-14T01:30:38.754640+0000","last_fullsized":"2023-10-14T01:30:38.754640+0000","mapping_epoch":15,"log_start":"133'567828","ondisk_log_start":"133'567828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7262949376,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":574324,"num_read_kb":2265116,"num_write":572784,"num_write_kb":2291136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3420,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:30:35.382221+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:30:35.382221+0000","last_peered":"2023-10-14T01:30:35.382221+0000","last_clean":"2023-10-14T01:30:35.382221+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:30:35.382221+0000","last_undegraded":"2023-10-14T01:30:35.382221+0000","last_fullsized":"2023-10-14T01:30:35.382221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'549556","reported_seq":1093642,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:38.424896+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:30:38.424896+0000","last_peered":"2023-10-14T01:30:38.424896+0000","last_clean":"2023-10-14T01:30:38.424896+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:30:38.424896+0000","last_undegraded":"2023-10-14T01:30:38.424896+0000","last_fullsized":"2023-10-14T01:30:38.424896+0000","mapping_epoch":15,"log_start":"133'539527","ondisk_log_start":"133'539527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6909366291,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":545855,"num_read_kb":2152060,"num_write":544732,"num_write_kb":2178922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'11113","reported_seq":15831,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:37.853939+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:30:37.853939+0000","last_peered":"2023-10-14T01:30:37.853939+0000","last_clean":"2023-10-14T01:30:37.853939+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:30:37.853939+0000","last_undegraded":"2023-10-14T01:30:37.853939+0000","last_fullsized":"2023-10-14T01:30:37.853939+0000","mapping_epoch":12,"log_start":"51'1100","ondisk_log_start":"51'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":315,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:43.050101+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:29:43.050101+0000","last_peered":"2023-10-14T01:29:43.050101+0000","last_clean":"2023-10-14T01:29:43.050101+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:29:43.050101+0000","last_undegraded":"2023-10-14T01:29:43.050101+0000","last_fullsized":"2023-10-14T01:29:43.050101+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28557152275,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2254370,"num_read_kb":8890244,"num_write":2249933,"num_write_kb":8999726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18428604416,"data_stored":27610968102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40214,"ondisk_log_size":40214,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49203,"ondisk_log_size":49203,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739396,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10984112,"kb_used_data":8996712,"kb_used_omap":1432,"kb_used_meta":1985959,"kb_avail":83387728,"statfs":{"total":96636764160,"available":85389033472,"internally_reserved":0,"allocated":9212633088,"data_stored":13802949461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":2033622990},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.38400000000000001,"15min":0.39500000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.3319999999999999,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.40400000000000003,"15min":0.41299999999999998},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.19600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:30:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.59599999999999997,"15min":0.51900000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82199999999999995,"5min":11.952,"15min":15.845000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.67700000000000005,"15min":0.59699999999999998},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":1.6319999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.31}]}]},{"osd":1,"up_from":8,"seq":34359739395,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6262196,"kb_used_data":4386728,"kb_used_omap":625,"kb_used_meta":1874830,"kb_avail":88109644,"statfs":{"total":96636764160,"available":90224275456,"internally_reserved":0,"allocated":4492009472,"data_stored":6732396458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1919826860},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,0,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51100000000000001,"15min":0.60799999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.399,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":3.488,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.46600000000000003,"15min":0.47899999999999998},"min":{"1min":0.214,"5min":0.193,"15min":0.185},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.41899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6123820,"kb_used_data":4616032,"kb_used_omap":746,"kb_used_meta":1507029,"kb_avail":88248020,"statfs":{"total":96636764160,"available":90365972480,"internally_reserved":0,"allocated":4726816768,"data_stored":7075606343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1543198432},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.63800000000000001,"15min":0.497},"min":{"1min":0.17699999999999999,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":5.2359999999999998,"5min":16.023,"15min":16.023},"last":0.313},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.48799999999999999},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":5.1909999999999998,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46100000000000002,"15min":0.499},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.4350000000000001,"5min":5.141,"15min":5.141},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.22700000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.1819999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4725284864,"data_stored":7075012608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4490436608,"data_stored":6731890707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9212882944,"data_stored":13804064787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:30:43.100 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:30:43.101 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:53.120 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.115+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:30:53.121 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.119+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:30:53.121 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.119+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:30:53.121 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.119+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:30:53.140 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.131+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:30:53.140 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.131+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:30:53.140 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.131+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:30:53.141 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.131+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:30:53.141 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.131+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:30:53.141 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.131+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:30:53.144 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:53.139+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:30:57.589 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:57.583+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:30:57.597 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:57.591+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:30:57.597 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:30:57.591+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:31:03.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:31:03.569 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:31:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:31:03.614 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2709,"stamp":"2023-10-14T01:31:01.655100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28589522661,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2296230,"num_read_kb":8951430,"num_write":2258846,"num_write_kb":9034443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89471,"ondisk_log_size":89471,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23542760,"kb_used_data":18064848,"kb_used_omap":2803,"kb_used_meta":5475084,"kb_avail":259572760,"statfs":{"total":289910292480,"available":265802506240,"internally_reserved":0,"allocated":18498404352,"data_stored":27677893143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870694,"internal_metadata":5606486618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,0,1,3,9,7,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15810560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4071,"num_read_kb":16256,"num_write":4058,"num_write_kb":16232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002088"},"pg_stats":[{"pgid":"2.7","version":"135'7533","reported_seq":10964,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.514755+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:30:58.514755+0000","last_peered":"2023-10-14T01:30:58.514755+0000","last_clean":"2023-10-14T01:30:58.514755+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:30:58.514755+0000","last_undegraded":"2023-10-14T01:30:58.514755+0000","last_fullsized":"2023-10-14T01:30:58.514755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7533,"log_dups_size":0,"ondisk_log_size":7533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'11224","reported_seq":17541,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.513170+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:30:58.513170+0000","last_peered":"2023-10-14T01:30:58.513170+0000","last_clean":"2023-10-14T01:30:58.513170+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:30:58.513170+0000","last_undegraded":"2023-10-14T01:30:58.513170+0000","last_fullsized":"2023-10-14T01:30:58.513170+0000","mapping_epoch":12,"log_start":"53'1200","ondisk_log_start":"53'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'7514","reported_seq":10744,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.508701+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:30:58.508701+0000","last_peered":"2023-10-14T01:30:58.508701+0000","last_clean":"2023-10-14T01:30:58.508701+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:30:58.508701+0000","last_undegraded":"2023-10-14T01:30:58.508701+0000","last_fullsized":"2023-10-14T01:30:58.508701+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'18518","reported_seq":27034,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.537882+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:30:58.537882+0000","last_peered":"2023-10-14T01:30:58.537882+0000","last_clean":"2023-10-14T01:30:58.537882+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:30:58.537882+0000","last_undegraded":"2023-10-14T01:30:58.537882+0000","last_fullsized":"2023-10-14T01:30:58.537882+0000","mapping_epoch":12,"log_start":"108'8500","ondisk_log_start":"108'8500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'588146","reported_seq":1170212,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.823764+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:30:58.823764+0000","last_peered":"2023-10-14T01:30:58.823764+0000","last_clean":"2023-10-14T01:30:58.823764+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:30:58.823764+0000","last_undegraded":"2023-10-14T01:30:58.823764+0000","last_fullsized":"2023-10-14T01:30:58.823764+0000","mapping_epoch":15,"log_start":"133'578136","ondisk_log_start":"133'578136","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7386198016,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":583966,"num_read_kb":2303472,"num_write":582982,"num_write_kb":2331928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"135'3901","reported_seq":5709,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:55.854926+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:30:55.854926+0000","last_peered":"2023-10-14T01:30:55.854926+0000","last_clean":"2023-10-14T01:30:55.854926+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:30:55.854926+0000","last_undegraded":"2023-10-14T01:30:55.854926+0000","last_fullsized":"2023-10-14T01:30:55.854926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'557667","reported_seq":1110390,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:30:58.578136+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:30:58.578136+0000","last_peered":"2023-10-14T01:30:58.578136+0000","last_clean":"2023-10-14T01:30:58.578136+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:30:58.578136+0000","last_undegraded":"2023-10-14T01:30:58.578136+0000","last_fullsized":"2023-10-14T01:30:58.578136+0000","mapping_epoch":15,"log_start":"133'547627","ondisk_log_start":"133'547627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2491,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 558s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7014809600,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":554404,"num_read_kb":2186292,"num_write":553555,"num_write_kb":2214220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3278,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:55.856285+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:30:55.856285+0000","last_peered":"2023-10-14T01:30:55.856285+0000","last_clean":"2023-10-14T01:30:55.856285+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:30:55.856285+0000","last_undegraded":"2023-10-14T01:30:55.856285+0000","last_fullsized":"2023-10-14T01:30:55.856285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'579860","reported_seq":1154359,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.792028+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:30:58.792028+0000","last_peered":"2023-10-14T01:30:58.792028+0000","last_clean":"2023-10-14T01:30:58.792028+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:30:58.792028+0000","last_undegraded":"2023-10-14T01:30:58.792028+0000","last_fullsized":"2023-10-14T01:30:58.792028+0000","mapping_epoch":15,"log_start":"133'569828","ondisk_log_start":"133'569828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7270797312,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":576349,"num_read_kb":2273180,"num_write":574798,"num_write_kb":2299192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3428,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:55.861089+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:30:55.861089+0000","last_peered":"2023-10-14T01:30:55.861089+0000","last_clean":"2023-10-14T01:30:55.861089+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:30:55.861089+0000","last_undegraded":"2023-10-14T01:30:55.861089+0000","last_fullsized":"2023-10-14T01:30:55.861089+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'551539","reported_seq":1097532,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.578187+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:30:58.578187+0000","last_peered":"2023-10-14T01:30:58.578187+0000","last_clean":"2023-10-14T01:30:58.578187+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:30:58.578187+0000","last_undegraded":"2023-10-14T01:30:58.578187+0000","last_fullsized":"2023-10-14T01:30:58.578187+0000","mapping_epoch":15,"log_start":"133'541527","ondisk_log_start":"133'541527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6917124115,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":547763,"num_read_kb":2159680,"num_write":546715,"num_write_kb":2186854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'11164","reported_seq":15894,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:30:58.527586+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:30:58.527586+0000","last_peered":"2023-10-14T01:30:58.527586+0000","last_clean":"2023-10-14T01:30:58.527586+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:30:58.527586+0000","last_undegraded":"2023-10-14T01:30:58.527586+0000","last_fullsized":"2023-10-14T01:30:58.527586+0000","mapping_epoch":12,"log_start":"51'1100","ondisk_log_start":"51'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":315,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:43.050101+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:29:43.050101+0000","last_peered":"2023-10-14T01:29:43.050101+0000","last_clean":"2023-10-14T01:29:43.050101+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:29:43.050101+0000","last_undegraded":"2023-10-14T01:29:43.050101+0000","last_fullsized":"2023-10-14T01:29:43.050101+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28588929043,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2262482,"num_read_kb":8922624,"num_write":2258050,"num_write_kb":9032194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18495152128,"data_stored":27677515814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40094,"ondisk_log_size":40094,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49278,"ondisk_log_size":49278,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11051968,"kb_used_data":9029496,"kb_used_omap":1432,"kb_used_meta":2021031,"kb_avail":83319872,"statfs":{"total":96636764160,"available":85319548928,"internally_reserved":0,"allocated":9246203904,"data_stored":13836516134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466418,"internal_metadata":2069536718},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.38400000000000001,"15min":0.39500000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.3319999999999999,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.183},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.40400000000000003,"15min":0.41299999999999998},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.22600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:30:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.59599999999999997,"15min":0.51900000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82199999999999995,"5min":11.952,"15min":15.845000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.67700000000000005,"15min":0.59699999999999998},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":1.6319999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.216}]}]},{"osd":1,"up_from":8,"seq":34359739399,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6296028,"kb_used_data":4402516,"kb_used_omap":625,"kb_used_meta":1892878,"kb_avail":88075812,"statfs":{"total":96636764160,"available":90189631488,"internally_reserved":0,"allocated":4508176384,"data_stored":6748563370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1938308012},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,3,9,7,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51100000000000001,"15min":0.60799999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.399,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":3.488,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.46600000000000003,"15min":0.47899999999999998},"min":{"1min":0.214,"5min":0.193,"15min":0.185},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6194764,"kb_used_data":4632836,"kb_used_omap":746,"kb_used_meta":1561173,"kb_avail":88177076,"statfs":{"total":96636764160,"available":90293325824,"internally_reserved":0,"allocated":4744024064,"data_stored":7092813639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1598641888},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.63800000000000001,"15min":0.497},"min":{"1min":0.17699999999999999,"5min":0.17399999999999999,"15min":0.17299999999999999},"max":{"1min":5.2359999999999998,"5min":16.023,"15min":16.023},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.48799999999999999},"min":{"1min":0.26100000000000001,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":5.1909999999999998,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46100000000000002,"15min":0.499},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.4350000000000001,"5min":5.141,"15min":5.141},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.435,"15min":0.43099999999999999},"min":{"1min":0.22700000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.1819999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4742275072,"data_stored":7092002816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4506701824,"data_stored":6748155923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9246175232,"data_stored":13837357075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:31:03.616 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:31:03.617 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:31:06.981 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:31:06.982 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:31:06.982 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:31:06.983 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:31:06.983 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:31:06.983 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:31:06.983 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:31:06.984 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:06.979+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:31:10.692 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.687+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:31:10.697 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.695+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:31:10.702 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:31:10.703 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:31:10.703 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:31:10.703 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:31:10.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:31:10.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:31:10.704 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:31:10.705 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:31:10.705 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:31:10.705 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.699+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:31:10.709 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.707+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:31:10.709 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:31:10.707+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:31:23.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:31:24.185 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:31:24.186 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:31:24.221 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2720,"stamp":"2023-10-14T01:31:23.658704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28621938405,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2304335,"num_read_kb":8983758,"num_write":2267196,"num_write_kb":9067843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89819,"ondisk_log_size":89819,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23713192,"kb_used_data":18139216,"kb_used_omap":2851,"kb_used_meta":5571100,"kb_avail":259402328,"statfs":{"total":289910292480,"available":265627983872,"internally_reserved":0,"allocated":18574557184,"data_stored":27754045975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2920193,"internal_metadata":5704806655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,2,9,13,9,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16330752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4006,"num_read_kb":16008,"num_write":4200,"num_write_kb":16800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001917"},"pg_stats":[{"pgid":"2.7","version":"135'7563","reported_seq":11002,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:16.335282+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:31:16.335282+0000","last_peered":"2023-10-14T01:31:16.335282+0000","last_clean":"2023-10-14T01:31:16.335282+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:31:16.335282+0000","last_undegraded":"2023-10-14T01:31:16.335282+0000","last_fullsized":"2023-10-14T01:31:16.335282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'11269","reported_seq":17598,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:16.335120+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:31:16.335120+0000","last_peered":"2023-10-14T01:31:16.335120+0000","last_clean":"2023-10-14T01:31:16.335120+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:31:16.335120+0000","last_undegraded":"2023-10-14T01:31:16.335120+0000","last_fullsized":"2023-10-14T01:31:16.335120+0000","mapping_epoch":12,"log_start":"53'1200","ondisk_log_start":"53'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'7544","reported_seq":10783,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:16.334943+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:31:16.334943+0000","last_peered":"2023-10-14T01:31:16.334943+0000","last_clean":"2023-10-14T01:31:16.334943+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:31:16.334943+0000","last_undegraded":"2023-10-14T01:31:16.334943+0000","last_fullsized":"2023-10-14T01:31:16.334943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'18593","reported_seq":27130,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:16.335468+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:31:16.335468+0000","last_peered":"2023-10-14T01:31:16.335468+0000","last_clean":"2023-10-14T01:31:16.335468+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:31:16.335468+0000","last_undegraded":"2023-10-14T01:31:16.335468+0000","last_fullsized":"2023-10-14T01:31:16.335468+0000","mapping_epoch":12,"log_start":"108'8500","ondisk_log_start":"108'8500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'590316","reported_seq":1174487,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:18.891275+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:31:18.891275+0000","last_peered":"2023-10-14T01:31:18.891275+0000","last_clean":"2023-10-14T01:31:18.891275+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:31:18.891275+0000","last_undegraded":"2023-10-14T01:31:18.891275+0000","last_fullsized":"2023-10-14T01:31:18.891275+0000","mapping_epoch":15,"log_start":"134'580236","ondisk_log_start":"134'580236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7394672640,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":586068,"num_read_kb":2311844,"num_write":585152,"num_write_kb":2340608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"135'3911","reported_seq":5724,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:16.406669+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:31:16.406669+0000","last_peered":"2023-10-14T01:31:16.406669+0000","last_clean":"2023-10-14T01:31:16.406669+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:31:16.406669+0000","last_undegraded":"2023-10-14T01:31:16.406669+0000","last_fullsized":"2023-10-14T01:31:16.406669+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'559765","reported_seq":1114485,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:31:18.710806+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:31:18.710806+0000","last_peered":"2023-10-14T01:31:18.710806+0000","last_clean":"2023-10-14T01:31:18.710806+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:31:18.710806+0000","last_undegraded":"2023-10-14T01:31:18.710806+0000","last_fullsized":"2023-10-14T01:31:18.710806+0000","mapping_epoch":15,"log_start":"134'549727","ondisk_log_start":"134'549727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2581,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 579s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7022710784,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":556388,"num_read_kb":2194216,"num_write":555590,"num_write_kb":2222360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3288,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:21.455279+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:31:21.455279+0000","last_peered":"2023-10-14T01:31:21.455279+0000","last_clean":"2023-10-14T01:31:21.455279+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:31:21.455279+0000","last_undegraded":"2023-10-14T01:31:21.455279+0000","last_fullsized":"2023-10-14T01:31:21.455279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'581917","reported_seq":1158491,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:18.894394+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:31:18.894394+0000","last_peered":"2023-10-14T01:31:18.894394+0000","last_clean":"2023-10-14T01:31:18.894394+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:31:18.894394+0000","last_undegraded":"2023-10-14T01:31:18.894394+0000","last_fullsized":"2023-10-14T01:31:18.894394+0000","mapping_epoch":15,"log_start":"134'571828","ondisk_log_start":"134'571828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7278682112,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":578427,"num_read_kb":2281476,"num_write":576855,"num_write_kb":2307420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3438,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:21.457487+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:31:21.457487+0000","last_peered":"2023-10-14T01:31:21.457487+0000","last_clean":"2023-10-14T01:31:21.457487+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:31:21.457487+0000","last_undegraded":"2023-10-14T01:31:21.457487+0000","last_fullsized":"2023-10-14T01:31:21.457487+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'553627","reported_seq":1101557,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:18.717064+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:31:18.717064+0000","last_peered":"2023-10-14T01:31:18.717064+0000","last_clean":"2023-10-14T01:31:18.717064+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:31:18.717064+0000","last_undegraded":"2023-10-14T01:31:18.717064+0000","last_fullsized":"2023-10-14T01:31:18.717064+0000","mapping_epoch":15,"log_start":"134'543527","ondisk_log_start":"134'543527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6925279251,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":549704,"num_read_kb":2167416,"num_write":548803,"num_write_kb":2195206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'11209","reported_seq":15952,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:16.335893+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:31:16.335893+0000","last_peered":"2023-10-14T01:31:16.335893+0000","last_clean":"2023-10-14T01:31:16.335893+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:31:16.335893+0000","last_undegraded":"2023-10-14T01:31:16.335893+0000","last_fullsized":"2023-10-14T01:31:16.335893+0000","mapping_epoch":12,"log_start":"53'1200","ondisk_log_start":"53'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":315,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:43.050101+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:29:43.050101+0000","last_peered":"2023-10-14T01:29:43.050101+0000","last_clean":"2023-10-14T01:29:43.050101+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:29:43.050101+0000","last_undegraded":"2023-10-14T01:29:43.050101+0000","last_fullsized":"2023-10-14T01:29:43.050101+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28621344787,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2270587,"num_read_kb":8954952,"num_write":2266400,"num_write_kb":9065594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18571423744,"data_stored":27753787430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40307,"ondisk_log_size":40307,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49413,"ondisk_log_size":49413,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739405,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11094508,"kb_used_data":9070372,"kb_used_omap":1480,"kb_used_meta":2022647,"kb_avail":83277332,"statfs":{"total":96636764160,"available":85275987968,"internally_reserved":0,"allocated":9288060928,"data_stored":13878373158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1515917,"internal_metadata":2071191155},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:31:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41799999999999998,"15min":0.40400000000000003},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.1369999999999996,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39500000000000002,"15min":0.41299999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.028,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:30:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.59599999999999997,"15min":0.51900000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82199999999999995,"5min":11.952,"15min":15.845000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.67700000000000005,"15min":0.59699999999999998},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":1.6319999999999999,"5min":12.029999999999999,"15min":15.805999999999999},"last":0.33500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739403,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6370836,"kb_used_data":4419020,"kb_used_omap":625,"kb_used_meta":1951182,"kb_avail":88001004,"statfs":{"total":96636764160,"available":90113028096,"internally_reserved":0,"allocated":4525076480,"data_stored":6765463466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1998011308},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,8,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51100000000000001,"15min":0.60799999999999998},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.399,"5min":6.1639999999999997,"15min":24.248999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.192},"max":{"1min":3.488,"5min":6.1989999999999998,"15min":17.263999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.46600000000000003,"15min":0.47899999999999998},"min":{"1min":0.214,"5min":0.193,"15min":0.185},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.376},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6247848,"kb_used_data":4649824,"kb_used_omap":746,"kb_used_meta":1597269,"kb_avail":88123992,"statfs":{"total":96636764160,"available":90238967808,"internally_reserved":0,"allocated":4761419776,"data_stored":7110209351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1635604192},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,8,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.63300000000000001,"15min":0.49299999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.54600000000000004,"5min":16.023,"15min":16.023},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.31,"5min":0.52200000000000002,"15min":0.47499999999999998},"min":{"1min":0.255,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":0.39800000000000002,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.41699999999999998,"15min":0.46999999999999997},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.66600000000000004,"5min":5.141,"15min":5.141},"last":0.221},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.41599999999999998,"15min":0.40500000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.48999999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4759588864,"data_stored":7109316608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4523601920,"data_stored":6765056019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9288232960,"data_stored":13879414803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:31:24.223 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:31:24.224 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:31:44.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:31:44.739 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:31:44.740 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:31:44.779 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2730,"stamp":"2023-10-14T01:31:43.662764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28653514469,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2312431,"num_read_kb":9016086,"num_write":2275336,"num_write_kb":9100403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89808,"ondisk_log_size":89808,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23976796,"kb_used_data":18205956,"kb_used_omap":2851,"kb_used_meta":5767964,"kb_avail":259138724,"statfs":{"total":289910292480,"available":265358053376,"internally_reserved":0,"allocated":18642898944,"data_stored":27822387735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2920193,"internal_metadata":5906395391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,4,7,14,18],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15974400,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4118,"num_read_kb":16436,"num_write":4150,"num_write_kb":16600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002094"},"pg_stats":[{"pgid":"2.7","version":"135'7599","reported_seq":11046,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:37.757508+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:31:37.757508+0000","last_peered":"2023-10-14T01:31:37.757508+0000","last_clean":"2023-10-14T01:31:37.757508+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:31:37.757508+0000","last_undegraded":"2023-10-14T01:31:37.757508+0000","last_fullsized":"2023-10-14T01:31:37.757508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7599,"log_dups_size":0,"ondisk_log_size":7599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'11323","reported_seq":17664,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:37.759679+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:31:37.759679+0000","last_peered":"2023-10-14T01:31:37.759679+0000","last_clean":"2023-10-14T01:31:37.759679+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:31:37.759679+0000","last_undegraded":"2023-10-14T01:31:37.759679+0000","last_fullsized":"2023-10-14T01:31:37.759679+0000","mapping_epoch":12,"log_start":"53'1300","ondisk_log_start":"53'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'7580","reported_seq":10827,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:37.747321+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:31:37.747321+0000","last_peered":"2023-10-14T01:31:37.747321+0000","last_clean":"2023-10-14T01:31:37.747321+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:31:37.747321+0000","last_undegraded":"2023-10-14T01:31:37.747321+0000","last_fullsized":"2023-10-14T01:31:37.747321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'18683","reported_seq":27240,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:37.790816+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:31:37.790816+0000","last_peered":"2023-10-14T01:31:37.790816+0000","last_clean":"2023-10-14T01:31:37.790816+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:31:37.790816+0000","last_undegraded":"2023-10-14T01:31:37.790816+0000","last_fullsized":"2023-10-14T01:31:37.790816+0000","mapping_epoch":12,"log_start":"110'8600","ondisk_log_start":"110'8600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'592495","reported_seq":1178733,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:39.051748+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:31:39.051748+0000","last_peered":"2023-10-14T01:31:39.051748+0000","last_clean":"2023-10-14T01:31:39.051748+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:31:39.051748+0000","last_undegraded":"2023-10-14T01:31:39.051748+0000","last_fullsized":"2023-10-14T01:31:39.051748+0000","mapping_epoch":15,"log_start":"134'582436","ondisk_log_start":"134'582436","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7403102208,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":588140,"num_read_kb":2320112,"num_write":587331,"num_write_kb":2349324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"135'3930","reported_seq":5747,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:38.596614+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:31:38.596614+0000","last_peered":"2023-10-14T01:31:38.596614+0000","last_clean":"2023-10-14T01:31:38.596614+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:31:38.596614+0000","last_undegraded":"2023-10-14T01:31:38.596614+0000","last_fullsized":"2023-10-14T01:31:38.596614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'561891","reported_seq":1118589,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:31:38.791057+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:31:38.791057+0000","last_peered":"2023-10-14T01:31:38.791057+0000","last_clean":"2023-10-14T01:31:38.791057+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:31:38.791057+0000","last_undegraded":"2023-10-14T01:31:38.791057+0000","last_fullsized":"2023-10-14T01:31:38.791057+0000","mapping_epoch":15,"log_start":"134'551827","ondisk_log_start":"134'551827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2672,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 599s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7030730752,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":558357,"num_read_kb":2202076,"num_write":557656,"num_write_kb":2230624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3296,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:41.936299+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:31:41.936299+0000","last_peered":"2023-10-14T01:31:41.936299+0000","last_clean":"2023-10-14T01:31:41.936299+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:31:41.936299+0000","last_undegraded":"2023-10-14T01:31:41.936299+0000","last_fullsized":"2023-10-14T01:31:41.936299+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'583872","reported_seq":1162528,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:38.996919+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:31:38.996919+0000","last_peered":"2023-10-14T01:31:38.996919+0000","last_clean":"2023-10-14T01:31:38.996919+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:31:38.996919+0000","last_undegraded":"2023-10-14T01:31:38.996919+0000","last_fullsized":"2023-10-14T01:31:38.996919+0000","mapping_epoch":15,"log_start":"134'573828","ondisk_log_start":"134'573828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7286206464,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":580505,"num_read_kb":2289780,"num_write":578810,"num_write_kb":2315240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3446,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:41.936388+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:31:41.936388+0000","last_peered":"2023-10-14T01:31:41.936388+0000","last_clean":"2023-10-14T01:31:41.936388+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:31:41.936388+0000","last_undegraded":"2023-10-14T01:31:41.936388+0000","last_fullsized":"2023-10-14T01:31:41.936388+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'555567","reported_seq":1105479,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:38.814033+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:31:38.814033+0000","last_peered":"2023-10-14T01:31:38.814033+0000","last_clean":"2023-10-14T01:31:38.814033+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:31:38.814033+0000","last_undegraded":"2023-10-14T01:31:38.814033+0000","last_fullsized":"2023-10-14T01:31:38.814033+0000","mapping_epoch":15,"log_start":"134'545527","ondisk_log_start":"134'545527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6932881427,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":551681,"num_read_kb":2175312,"num_write":550743,"num_write_kb":2202966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"135'11263","reported_seq":16018,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:37.812602+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:31:37.812602+0000","last_peered":"2023-10-14T01:31:37.812602+0000","last_clean":"2023-10-14T01:31:37.812602+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:31:37.812602+0000","last_undegraded":"2023-10-14T01:31:37.812602+0000","last_fullsized":"2023-10-14T01:31:37.812602+0000","mapping_epoch":12,"log_start":"53'1200","ondisk_log_start":"53'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":315,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:29:43.050101+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:29:43.050101+0000","last_peered":"2023-10-14T01:29:43.050101+0000","last_clean":"2023-10-14T01:29:43.050101+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:29:43.050101+0000","last_undegraded":"2023-10-14T01:29:43.050101+0000","last_fullsized":"2023-10-14T01:29:43.050101+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28652920851,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2278683,"num_read_kb":8987280,"num_write":2274540,"num_write_kb":9098154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18638450688,"data_stored":27820814374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49502,"ondisk_log_size":49502,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739409,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11236972,"kb_used_data":9104612,"kb_used_omap":1480,"kb_used_meta":2130871,"kb_avail":83134868,"statfs":{"total":96636764160,"available":85130104832,"internally_reserved":0,"allocated":9323122688,"data_stored":13913434918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1515917,"internal_metadata":2182012531},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:31:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41799999999999998,"15min":0.40400000000000003},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.1369999999999996,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.224},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39500000000000002,"15min":0.41299999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.028,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.30299999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.433,"15min":0.51700000000000002},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58199999999999996,"5min":4.8150000000000004,"15min":15.845000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.51300000000000001,"15min":0.59299999999999997},"min":{"1min":0.187,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.67500000000000004,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739407,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6439572,"kb_used_data":4435212,"kb_used_omap":625,"kb_used_meta":2003726,"kb_avail":87932268,"statfs":{"total":96636764160,"available":90042642432,"internally_reserved":0,"allocated":4541657088,"data_stored":6782044074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":2051816364},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,5,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.39100000000000001,"15min":0.58399999999999996},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":0.64300000000000002,"5min":3.399,"15min":24.248999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.44700000000000001,"15min":0.56399999999999995},"min":{"1min":0.248,"5min":0.248,"15min":0.192},"max":{"1min":0.68500000000000005,"5min":3.488,"15min":17.263999999999999},"last":0.315}]},{"osd":2,"last update":"Sat Oct 14 01:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.46700000000000003,"15min":0.47999999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.185},"max":{"1min":0.62,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.251},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.496,"15min":0.57799999999999996},"min":{"1min":0.215,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.438,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6300252,"kb_used_data":4666132,"kb_used_omap":746,"kb_used_meta":1633365,"kb_avail":88071588,"statfs":{"total":96636764160,"available":90185306112,"internally_reserved":0,"allocated":4778119168,"data_stored":7126908743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1672566496},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.63300000000000001,"15min":0.49299999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.54600000000000004,"5min":16.023,"15min":16.023},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.52200000000000002,"15min":0.47499999999999998},"min":{"1min":0.255,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":0.39800000000000002,"5min":16.065000000000001,"15min":16.065000000000001},"last":0.19800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.41699999999999998,"15min":0.46999999999999997},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.66600000000000004,"5min":5.141,"15min":5.141},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.41599999999999998,"15min":0.40500000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.48999999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4776529920,"data_stored":7126257664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4540014592,"data_stored":6781468691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9321906176,"data_stored":13913088019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:31:44.780 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:31:44.781 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:31:57.451 INFO:tasks.ceph.osd.0.smithi081.stderr:2023-10-14T01:31:57.450+0000 7f3602a3d640 -1 reset not still connected to 0x55a8b615d6c0 2023-10-14T01:32:00.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:32:00.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:32:00.199 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:32:00.200 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.198+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:32:00.202 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.202+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:32:00.202 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.202+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:32:00.203 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.202+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:32:00.213 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:00.210+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:32:04.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:32:05.025 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.014+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:32:05.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.018+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:32:05.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.018+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:32:05.026 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.018+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:32:05.027 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.018+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:32:05.030 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.030+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:32:05.049 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:05.050+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:32:05.792 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:32:05.793 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:32:05.827 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2742,"stamp":"2023-10-14T01:32:05.666450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28694269669,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2322694,"num_read_kb":9057050,"num_write":2285840,"num_write_kb":9142419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89866,"ondisk_log_size":89866,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22404620,"kb_used_data":18281968,"kb_used_omap":2853,"kb_used_meta":4119770,"kb_avail":260710900,"statfs":{"total":289910292480,"available":266967961600,"internally_reserved":0,"allocated":18720735232,"data_stored":27900211465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2921873,"internal_metadata":4218645103},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,12,12,6,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24600576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6193,"num_read_kb":24736,"num_write":6332,"num_write_kb":25328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001998"},"pg_stats":[{"pgid":"2.7","version":"136'7635","reported_seq":11093,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:03.610684+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:32:03.610684+0000","last_peered":"2023-10-14T01:32:03.610684+0000","last_clean":"2023-10-14T01:32:03.610684+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:32:03.610684+0000","last_undegraded":"2023-10-14T01:32:03.610684+0000","last_fullsized":"2023-10-14T01:32:03.610684+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'11376","reported_seq":17733,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:03.631946+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:32:03.631946+0000","last_peered":"2023-10-14T01:32:03.631946+0000","last_clean":"2023-10-14T01:32:03.631946+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:32:03.631946+0000","last_undegraded":"2023-10-14T01:32:03.631946+0000","last_fullsized":"2023-10-14T01:32:03.631946+0000","mapping_epoch":12,"log_start":"53'1300","ondisk_log_start":"53'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'7616","reported_seq":10874,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:03.601565+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:32:03.601565+0000","last_peered":"2023-10-14T01:32:03.601565+0000","last_clean":"2023-10-14T01:32:03.601565+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:32:03.601565+0000","last_undegraded":"2023-10-14T01:32:03.601565+0000","last_fullsized":"2023-10-14T01:32:03.601565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7616,"log_dups_size":0,"ondisk_log_size":7616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'18770","reported_seq":27353,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:03.649726+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:32:03.649726+0000","last_peered":"2023-10-14T01:32:03.649726+0000","last_clean":"2023-10-14T01:32:03.649726+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:32:03.649726+0000","last_undegraded":"2023-10-14T01:32:03.649726+0000","last_fullsized":"2023-10-14T01:32:03.649726+0000","mapping_epoch":12,"log_start":"110'8700","ondisk_log_start":"110'8700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'595167","reported_seq":1184139,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:04.131988+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:32:04.131988+0000","last_peered":"2023-10-14T01:32:04.131988+0000","last_clean":"2023-10-14T01:32:04.131988+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:32:04.131988+0000","last_undegraded":"2023-10-14T01:32:04.131988+0000","last_fullsized":"2023-10-14T01:32:04.131988+0000","mapping_epoch":15,"log_start":"134'585136","ondisk_log_start":"134'585136","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7413481472,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":590869,"num_read_kb":2331008,"num_write":590003,"num_write_kb":2360012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'3950","reported_seq":5773,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:02.414967+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:32:02.414967+0000","last_peered":"2023-10-14T01:32:02.414967+0000","last_clean":"2023-10-14T01:32:02.414967+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:32:02.414967+0000","last_undegraded":"2023-10-14T01:32:02.414967+0000","last_fullsized":"2023-10-14T01:32:02.414967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3950,"log_dups_size":0,"ondisk_log_size":3950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'564583","reported_seq":1123795,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:32:03.965623+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:32:03.965623+0000","last_peered":"2023-10-14T01:32:03.965623+0000","last_clean":"2023-10-14T01:32:03.965623+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:32:03.965623+0000","last_undegraded":"2023-10-14T01:32:03.965623+0000","last_fullsized":"2023-10-14T01:32:03.965623+0000","mapping_epoch":15,"log_start":"134'554527","ondisk_log_start":"134'554527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2771,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 624s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7041003520,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":560861,"num_read_kb":2212080,"num_write":560278,"num_write_kb":2241112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3303,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:31:57.302957+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:31:57.302957+0000","last_peered":"2023-10-14T01:31:57.302957+0000","last_clean":"2023-10-14T01:31:57.302957+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:31:57.302957+0000","last_undegraded":"2023-10-14T01:31:57.302957+0000","last_fullsized":"2023-10-14T01:31:57.302957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'586555","reported_seq":1167757,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:04.124528+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:32:04.124528+0000","last_peered":"2023-10-14T01:32:04.124528+0000","last_clean":"2023-10-14T01:32:04.124528+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:32:04.124528+0000","last_undegraded":"2023-10-14T01:32:04.124528+0000","last_fullsized":"2023-10-14T01:32:04.124528+0000","mapping_epoch":15,"log_start":"134'576528","ondisk_log_start":"134'576528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7296487424,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":583050,"num_read_kb":2299940,"num_write":581493,"num_write_kb":2325972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3454,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:31:57.303461+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:31:57.303461+0000","last_peered":"2023-10-14T01:31:57.303461+0000","last_clean":"2023-10-14T01:31:57.303461+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:31:57.303461+0000","last_undegraded":"2023-10-14T01:31:57.303461+0000","last_fullsized":"2023-10-14T01:31:57.303461+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'558094","reported_seq":1110493,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:03.948213+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:32:03.948213+0000","last_peered":"2023-10-14T01:32:03.948213+0000","last_clean":"2023-10-14T01:32:03.948213+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:32:03.948213+0000","last_undegraded":"2023-10-14T01:32:03.948213+0000","last_fullsized":"2023-10-14T01:32:03.948213+0000","mapping_epoch":15,"log_start":"134'548027","ondisk_log_start":"134'548027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6942703635,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":554166,"num_read_kb":2185216,"num_write":553270,"num_write_kb":2213074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'11315","reported_seq":16086,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:03.657471+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:32:03.657471+0000","last_peered":"2023-10-14T01:32:03.657471+0000","last_clean":"2023-10-14T01:32:03.657471+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:32:03.657471+0000","last_undegraded":"2023-10-14T01:32:03.657471+0000","last_fullsized":"2023-10-14T01:32:03.657471+0000","mapping_epoch":12,"log_start":"53'1300","ondisk_log_start":"53'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":317,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:43.855112+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:31:43.855112+0000","last_peered":"2023-10-14T01:31:43.855112+0000","last_clean":"2023-10-14T01:31:43.855112+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:31:43.855112+0000","last_undegraded":"2023-10-14T01:31:43.855112+0000","last_fullsized":"2023-10-14T01:31:43.855112+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28693676051,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2288946,"num_read_kb":9028244,"num_write":2285044,"num_write_kb":9140170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18715885568,"data_stored":27898249254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49586,"ondisk_log_size":49586,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739413,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9945280,"kb_used_data":9138296,"kb_used_omap":1482,"kb_used_meta":805493,"kb_avail":84426560,"statfs":{"total":96636764160,"available":86452797440,"internally_reserved":0,"allocated":9357615104,"data_stored":13947923148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":824825827},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:31:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41799999999999998,"15min":0.40400000000000003},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.1369999999999996,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39500000000000002,"15min":0.41299999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.028,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.433,"15min":0.51700000000000002},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58199999999999996,"5min":4.8150000000000004,"15min":15.845000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.51300000000000001,"15min":0.59299999999999997},"min":{"1min":0.187,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.67500000000000004,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.45400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739412,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6067176,"kb_used_data":4455900,"kb_used_omap":625,"kb_used_meta":1610638,"kb_avail":88304664,"statfs":{"total":96636764160,"available":90423975936,"internally_reserved":0,"allocated":4562841600,"data_stored":6803224400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1649294252},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,3,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.39100000000000001,"15min":0.58399999999999996},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":0.64300000000000002,"5min":3.399,"15min":24.248999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.44700000000000001,"15min":0.56399999999999995},"min":{"1min":0.248,"5min":0.248,"15min":0.192},"max":{"1min":0.68500000000000005,"5min":3.488,"15min":17.263999999999999},"last":0.499}]},{"osd":2,"last update":"Sat Oct 14 01:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.46700000000000003,"15min":0.47999999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.185},"max":{"1min":0.62,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.496,"15min":0.57799999999999996},"min":{"1min":0.215,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.438,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6392164,"kb_used_data":4687772,"kb_used_omap":746,"kb_used_meta":1703637,"kb_avail":87979676,"statfs":{"total":96636764160,"available":90091188224,"internally_reserved":0,"allocated":4800278528,"data_stored":7149063917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1744525024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.63300000000000001,"15min":0.49299999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.54600000000000004,"5min":16.023,"15min":16.023},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.52200000000000002,"15min":0.47499999999999998},"min":{"1min":0.255,"5min":0.224,"15min":0.17899999999999999},"max":{"1min":0.39800000000000002,"5min":16.065000000000001,"15min":16.065000000000001},"last":4.2039999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.41699999999999998,"15min":0.46999999999999997},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.66600000000000004,"5min":5.141,"15min":5.141},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.41599999999999998,"15min":0.40500000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.48999999999999999,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":4.274}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4798447616,"data_stored":7148175360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4561088512,"data_stored":6802542611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9356349440,"data_stored":13947531283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:32:05.829 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:32:05.829 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:32:17.700 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.697+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf0c820 2023-10-14T01:32:17.700 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.697+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:32:17.714 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.701+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:32:17.715 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.701+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d4cb790 2023-10-14T01:32:17.715 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.701+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:32:17.715 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.701+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:32:17.715 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.701+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:32:17.716 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.701+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd930 2023-10-14T01:32:17.716 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:32:17.716 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:32:17.716 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:32:17.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:32:17.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1aaf70 2023-10-14T01:32:17.717 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:32:17.718 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:17.713+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e625e10 2023-10-14T01:32:25.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:32:26.408 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:32:26.408 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:32:26.443 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2752,"stamp":"2023-10-14T01:32:25.669773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28727144165,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2330965,"num_read_kb":9090042,"num_write":2294284,"num_write_kb":9176195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89725,"ondisk_log_size":89725,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22658824,"kb_used_data":18349360,"kb_used_omap":2853,"kb_used_meta":4306586,"kb_avail":260456696,"statfs":{"total":289910292480,"available":266707656704,"internally_reserved":0,"allocated":18789744640,"data_stored":27969220873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2921873,"internal_metadata":4409944687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,2,3,10,11,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24670208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6223,"num_read_kb":24832,"num_write":6345,"num_write_kb":25380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001942"},"pg_stats":[{"pgid":"2.7","version":"136'7653","reported_seq":11120,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:23.215023+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:32:23.215023+0000","last_peered":"2023-10-14T01:32:23.215023+0000","last_clean":"2023-10-14T01:32:23.215023+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:32:23.215023+0000","last_undegraded":"2023-10-14T01:32:23.215023+0000","last_fullsized":"2023-10-14T01:32:23.215023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'11403","reported_seq":17773,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:23.241328+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:32:23.241328+0000","last_peered":"2023-10-14T01:32:23.241328+0000","last_clean":"2023-10-14T01:32:23.241328+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:32:23.241328+0000","last_undegraded":"2023-10-14T01:32:23.241328+0000","last_fullsized":"2023-10-14T01:32:23.241328+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'7634","reported_seq":10900,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:23.214915+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:32:23.214915+0000","last_peered":"2023-10-14T01:32:23.214915+0000","last_clean":"2023-10-14T01:32:23.214915+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:32:23.214915+0000","last_undegraded":"2023-10-14T01:32:23.214915+0000","last_fullsized":"2023-10-14T01:32:23.214915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'18815","reported_seq":27418,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:23.250620+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:32:23.250620+0000","last_peered":"2023-10-14T01:32:23.250620+0000","last_clean":"2023-10-14T01:32:23.250620+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:32:23.250620+0000","last_undegraded":"2023-10-14T01:32:23.250620+0000","last_fullsized":"2023-10-14T01:32:23.250620+0000","mapping_epoch":12,"log_start":"112'8800","ondisk_log_start":"112'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'597341","reported_seq":1188498,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:24.243257+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:32:24.243257+0000","last_peered":"2023-10-14T01:32:24.243257+0000","last_clean":"2023-10-14T01:32:24.243257+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:32:24.243257+0000","last_undegraded":"2023-10-14T01:32:24.243257+0000","last_fullsized":"2023-10-14T01:32:24.243257+0000","mapping_epoch":15,"log_start":"135'587336","ondisk_log_start":"135'587336","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7422033920,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":593055,"num_read_kb":2339728,"num_write":592177,"num_write_kb":2368708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'3964","reported_seq":5791,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:21.119136+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:32:21.119136+0000","last_peered":"2023-10-14T01:32:21.119136+0000","last_clean":"2023-10-14T01:32:21.119136+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:32:21.119136+0000","last_undegraded":"2023-10-14T01:32:21.119136+0000","last_fullsized":"2023-10-14T01:32:21.119136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'566739","reported_seq":1127931,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:32:24.086439+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:32:24.086439+0000","last_peered":"2023-10-14T01:32:24.086439+0000","last_clean":"2023-10-14T01:32:24.086439+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:32:24.086439+0000","last_undegraded":"2023-10-14T01:32:24.086439+0000","last_fullsized":"2023-10-14T01:32:24.086439+0000","mapping_epoch":15,"log_start":"135'556728","ondisk_log_start":"135'556728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2866,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 644s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7049175040,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":562828,"num_read_kb":2219924,"num_write":562367,"num_write_kb":2249468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3311,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:17.781077+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:32:17.781077+0000","last_peered":"2023-10-14T01:32:17.781077+0000","last_clean":"2023-10-14T01:32:17.781077+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:32:17.781077+0000","last_undegraded":"2023-10-14T01:32:17.781077+0000","last_fullsized":"2023-10-14T01:32:17.781077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'588685","reported_seq":1171991,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:24.219095+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:32:24.219095+0000","last_peered":"2023-10-14T01:32:24.219095+0000","last_clean":"2023-10-14T01:32:24.219095+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:32:24.219095+0000","last_undegraded":"2023-10-14T01:32:24.219095+0000","last_fullsized":"2023-10-14T01:32:24.219095+0000","mapping_epoch":15,"log_start":"135'578628","ondisk_log_start":"135'578628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7304650752,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":585160,"num_read_kb":2308356,"num_write":583623,"num_write_kb":2334492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3462,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:17.782764+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:32:17.782764+0000","last_peered":"2023-10-14T01:32:17.782764+0000","last_clean":"2023-10-14T01:32:17.782764+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:32:17.782764+0000","last_undegraded":"2023-10-14T01:32:17.782764+0000","last_fullsized":"2023-10-14T01:32:17.782764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'560145","reported_seq":1114552,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:24.088444+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:32:24.088444+0000","last_peered":"2023-10-14T01:32:24.088444+0000","last_clean":"2023-10-14T01:32:24.088444+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:32:24.088444+0000","last_undegraded":"2023-10-14T01:32:24.088444+0000","last_fullsized":"2023-10-14T01:32:24.088444+0000","mapping_epoch":15,"log_start":"135'550127","ondisk_log_start":"135'550127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6950690835,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":556174,"num_read_kb":2193228,"num_write":555321,"num_write_kb":2221278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'11342","reported_seq":16125,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:23.238950+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:32:23.238950+0000","last_peered":"2023-10-14T01:32:23.238950+0000","last_clean":"2023-10-14T01:32:23.238950+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:32:23.238950+0000","last_undegraded":"2023-10-14T01:32:23.238950+0000","last_fullsized":"2023-10-14T01:32:23.238950+0000","mapping_epoch":12,"log_start":"53'1300","ondisk_log_start":"53'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":317,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:43.855112+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:31:43.855112+0000","last_peered":"2023-10-14T01:31:43.855112+0000","last_clean":"2023-10-14T01:31:43.855112+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:31:43.855112+0000","last_undegraded":"2023-10-14T01:31:43.855112+0000","last_fullsized":"2023-10-14T01:31:43.855112+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28726550547,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2297217,"num_read_kb":9061236,"num_write":2293488,"num_write_kb":9173946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18784940032,"data_stored":27967303718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40091,"ondisk_log_size":40091,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49535,"ondisk_log_size":49535,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739417,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10069088,"kb_used_data":9171928,"kb_used_omap":1482,"kb_used_meta":895669,"kb_avail":84302752,"statfs":{"total":96636764160,"available":86326018048,"internally_reserved":0,"allocated":9392054272,"data_stored":13982362316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":917166051},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.42999999999999999,"15min":0.39100000000000001},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":0.77700000000000002,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.39700000000000002,"15min":0.39400000000000002},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.22900000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.433,"15min":0.51700000000000002},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58199999999999996,"5min":4.8150000000000004,"15min":15.845000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.51300000000000001,"15min":0.59299999999999997},"min":{"1min":0.187,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.67500000000000004,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.214}]}]},{"osd":1,"up_from":8,"seq":34359739416,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6137808,"kb_used_data":4472392,"kb_used_omap":625,"kb_used_meta":1664782,"kb_avail":88234032,"statfs":{"total":96636764160,"available":90351648768,"internally_reserved":0,"allocated":4579729408,"data_stored":6820112208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1704737708},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,9,11,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38800000000000001,"15min":0.57699999999999996},"min":{"1min":0.22900000000000001,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":0.48599999999999999,"5min":3.399,"15min":24.248999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.442,"15min":0.56299999999999994},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.192},"max":{"1min":0.53500000000000003,"5min":3.488,"15min":17.263999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.41499999999999998,"15min":0.44900000000000001},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.50700000000000001,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.48799999999999999,"15min":0.57199999999999995},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.436,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.28100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6451928,"kb_used_data":4705040,"kb_used_omap":746,"kb_used_meta":1746133,"kb_avail":87919912,"statfs":{"total":96636764160,"available":90029989888,"internally_reserved":0,"allocated":4817960960,"data_stored":7166746349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1788040928},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.432,"15min":0.48899999999999999},"min":{"1min":0.18099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.56299999999999994,"5min":5.2359999999999998,"15min":16.023},"last":5.7549999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.44600000000000001,"15min":0.48899999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.17899999999999999},"max":{"1min":4.6360000000000001,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.224}]},{"osd":2,"last update":"Sat Oct 14 01:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.439,"15min":0.46300000000000002},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.8069999999999999,"5min":5.141,"15min":5.141},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.51300000000000001,"15min":0.434},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":4.673,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4816089088,"data_stored":7165816832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4578054144,"data_stored":6819508243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9390796800,"data_stored":13981978643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:32:26.444 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:32:26.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:32:26.953 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:32:26.997 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:32:45.832 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:45.825+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:32:45.833 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:45.829+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:32:46.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:32:47.110 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:47.101+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:32:47.111 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:47.101+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:32:47.116 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:47.113+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:32:47.441 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:32:47.442 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:32:47.479 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2762,"stamp":"2023-10-14T01:32:45.674921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28759781093,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2339051,"num_read_kb":9122270,"num_write":2302736,"num_write_kb":9210003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89896,"ondisk_log_size":89896,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22904304,"kb_used_data":18417176,"kb_used_omap":2853,"kb_used_meta":4484250,"kb_avail":260211216,"statfs":{"total":289910292480,"available":266456285184,"internally_reserved":0,"allocated":18859188224,"data_stored":28038664457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2921873,"internal_metadata":4591872623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,7,13,12,19,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24526848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6038,"num_read_kb":24072,"num_write":6353,"num_write_kb":25412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003800"},"pg_stats":[{"pgid":"2.7","version":"136'7673","reported_seq":11148,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:43.375363+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:32:43.375363+0000","last_peered":"2023-10-14T01:32:43.375363+0000","last_clean":"2023-10-14T01:32:43.375363+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:32:43.375363+0000","last_undegraded":"2023-10-14T01:32:43.375363+0000","last_fullsized":"2023-10-14T01:32:43.375363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'11433","reported_seq":17815,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:43.375160+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:32:43.375160+0000","last_peered":"2023-10-14T01:32:43.375160+0000","last_clean":"2023-10-14T01:32:43.375160+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:32:43.375160+0000","last_undegraded":"2023-10-14T01:32:43.375160+0000","last_fullsized":"2023-10-14T01:32:43.375160+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'7654","reported_seq":10928,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:43.375468+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:32:43.375468+0000","last_peered":"2023-10-14T01:32:43.375468+0000","last_clean":"2023-10-14T01:32:43.375468+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:32:43.375468+0000","last_undegraded":"2023-10-14T01:32:43.375468+0000","last_fullsized":"2023-10-14T01:32:43.375468+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'18865","reported_seq":27488,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:43.375624+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:32:43.375624+0000","last_peered":"2023-10-14T01:32:43.375624+0000","last_clean":"2023-10-14T01:32:43.375624+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:32:43.375624+0000","last_undegraded":"2023-10-14T01:32:43.375624+0000","last_fullsized":"2023-10-14T01:32:43.375624+0000","mapping_epoch":12,"log_start":"112'8800","ondisk_log_start":"112'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'599489","reported_seq":1192814,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:44.293953+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:32:44.293953+0000","last_peered":"2023-10-14T01:32:44.293953+0000","last_clean":"2023-10-14T01:32:44.293953+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:32:44.293953+0000","last_undegraded":"2023-10-14T01:32:44.293953+0000","last_fullsized":"2023-10-14T01:32:44.293953+0000","mapping_epoch":15,"log_start":"135'589436","ondisk_log_start":"135'589436","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7430324224,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":595222,"num_read_kb":2348356,"num_write":594325,"num_write_kb":2377300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'3976","reported_seq":5807,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:43.375167+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:32:43.375167+0000","last_peered":"2023-10-14T01:32:43.375167+0000","last_clean":"2023-10-14T01:32:43.375167+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:32:43.375167+0000","last_undegraded":"2023-10-14T01:32:43.375167+0000","last_fullsized":"2023-10-14T01:32:43.375167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'568829","reported_seq":1131967,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:32:44.210053+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:32:44.210053+0000","last_peered":"2023-10-14T01:32:44.210053+0000","last_clean":"2023-10-14T01:32:44.210053+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:32:44.210053+0000","last_undegraded":"2023-10-14T01:32:44.210053+0000","last_fullsized":"2023-10-14T01:32:44.210053+0000","mapping_epoch":15,"log_start":"135'558828","ondisk_log_start":"135'558828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":2952,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 664s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7057068032,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":564764,"num_read_kb":2227636,"num_write":564400,"num_write_kb":2257600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3319,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:38.256595+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:32:38.256595+0000","last_peered":"2023-10-14T01:32:38.256595+0000","last_clean":"2023-10-14T01:32:38.256595+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:32:38.256595+0000","last_undegraded":"2023-10-14T01:32:38.256595+0000","last_fullsized":"2023-10-14T01:32:38.256595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'590849","reported_seq":1176208,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:44.273110+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:32:44.273110+0000","last_peered":"2023-10-14T01:32:44.273110+0000","last_clean":"2023-10-14T01:32:44.273110+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:32:44.273110+0000","last_undegraded":"2023-10-14T01:32:44.273110+0000","last_fullsized":"2023-10-14T01:32:44.273110+0000","mapping_epoch":15,"log_start":"135'580828","ondisk_log_start":"135'580828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7312912384,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":587209,"num_read_kb":2316532,"num_write":585787,"num_write_kb":2343148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3470,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:38.277446+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:32:38.277446+0000","last_peered":"2023-10-14T01:32:38.277446+0000","last_clean":"2023-10-14T01:32:38.277446+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:32:38.277446+0000","last_undegraded":"2023-10-14T01:32:38.277446+0000","last_fullsized":"2023-10-14T01:32:38.277446+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'562252","reported_seq":1118593,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:44.215227+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:32:44.215227+0000","last_peered":"2023-10-14T01:32:44.215227+0000","last_clean":"2023-10-14T01:32:44.215227+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:32:44.215227+0000","last_undegraded":"2023-10-14T01:32:44.215227+0000","last_fullsized":"2023-10-14T01:32:44.215227+0000","mapping_epoch":15,"log_start":"135'552227","ondisk_log_start":"135'552227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6958882835,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":558108,"num_read_kb":2200940,"num_write":557428,"num_write_kb":2229706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'11372","reported_seq":16167,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:32:43.375331+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:32:43.375331+0000","last_peered":"2023-10-14T01:32:43.375331+0000","last_clean":"2023-10-14T01:32:43.375331+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:32:43.375331+0000","last_undegraded":"2023-10-14T01:32:43.375331+0000","last_fullsized":"2023-10-14T01:32:43.375331+0000","mapping_epoch":12,"log_start":"53'1300","ondisk_log_start":"53'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":317,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:43.855112+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:31:43.855112+0000","last_peered":"2023-10-14T01:31:43.855112+0000","last_clean":"2023-10-14T01:31:43.855112+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:31:43.855112+0000","last_undegraded":"2023-10-14T01:31:43.855112+0000","last_fullsized":"2023-10-14T01:31:43.855112+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28759187475,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2305303,"num_read_kb":9093464,"num_write":2301940,"num_write_kb":9207754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18854031360,"data_stored":28036395046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40100,"ondisk_log_size":40100,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49697,"ondisk_log_size":49697,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739421,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10220468,"kb_used_data":9205356,"kb_used_omap":1482,"kb_used_meta":1013621,"kb_avail":84151372,"statfs":{"total":96636764160,"available":86171004928,"internally_reserved":0,"allocated":9426284544,"data_stored":14016592588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1037948899},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.42999999999999999,"15min":0.39100000000000001},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":0.77700000000000002,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.39700000000000002,"15min":0.39400000000000002},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:32:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.38400000000000001,"15min":0.50600000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52700000000000002,"5min":4.8150000000000004,"15min":15.845000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48099999999999998,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.81399999999999995,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739420,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6178488,"kb_used_data":4489456,"kb_used_omap":625,"kb_used_meta":1688398,"kb_avail":88193352,"statfs":{"total":96636764160,"available":90309992448,"internally_reserved":0,"allocated":4597202944,"data_stored":6837585744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1728920492},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,4,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38800000000000001,"15min":0.57699999999999996},"min":{"1min":0.22900000000000001,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":0.48599999999999999,"5min":3.399,"15min":24.248999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.442,"15min":0.56299999999999994},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.192},"max":{"1min":0.53500000000000003,"5min":3.488,"15min":17.263999999999999},"last":0.318}]},{"osd":2,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.41499999999999998,"15min":0.44900000000000001},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.50700000000000001,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.48799999999999999,"15min":0.57199999999999995},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.436,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.26500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6505348,"kb_used_data":4722364,"kb_used_omap":746,"kb_used_meta":1782229,"kb_avail":87866492,"statfs":{"total":96636764160,"available":89975287808,"internally_reserved":0,"allocated":4835700736,"data_stored":7184486125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":764192,"internal_metadata":1825003232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.432,"15min":0.48899999999999999},"min":{"1min":0.18099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.56299999999999994,"5min":5.2359999999999998,"15min":16.023},"last":1.4299999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.44600000000000001,"15min":0.48899999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.17899999999999999},"max":{"1min":4.6360000000000001,"5min":5.1909999999999998,"15min":16.065000000000001},"last":1.4179999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.439,"15min":0.46300000000000002},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.8069999999999999,"5min":5.141,"15min":5.141},"last":1.5129999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.51300000000000001,"15min":0.434},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":4.673,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4833742848,"data_stored":7183470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4595003392,"data_stored":6836457491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9425285120,"data_stored":14016466963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:32:47.481 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:32:47.481 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:32:53.458 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:53.453+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:32:53.460 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:32:53.457+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:33:04.505 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:04.484+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:33:04.505 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:04.492+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:33:04.506 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:04.492+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:33:04.506 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:04.496+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:33:04.507 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:04.496+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:33:07.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:33:07.997 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:33:07.998 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:33:08.020 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2774,"stamp":"2023-10-14T01:33:07.678852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28791639781,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2347002,"num_read_kb":9153966,"num_write":2311002,"num_write_kb":9243067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90053,"ondisk_log_size":90053,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22236820,"kb_used_data":18491704,"kb_used_omap":2855,"kb_used_meta":3742232,"kb_avail":260878700,"statfs":{"total":289910292480,"available":267139788800,"internally_reserved":0,"allocated":18935504896,"data_stored":28114976900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2923553,"internal_metadata":3832046559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,2,9,17,22,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15880192,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3937,"num_read_kb":15688,"num_write":4126,"num_write_kb":16504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001849"},"pg_stats":[{"pgid":"2.7","version":"136'7713","reported_seq":11196,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.345163+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:33:04.345163+0000","last_peered":"2023-10-14T01:33:04.345163+0000","last_clean":"2023-10-14T01:33:04.345163+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:33:04.345163+0000","last_undegraded":"2023-10-14T01:33:04.345163+0000","last_fullsized":"2023-10-14T01:33:04.345163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'11496","reported_seq":17890,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.345476+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:33:04.345476+0000","last_peered":"2023-10-14T01:33:04.345476+0000","last_clean":"2023-10-14T01:33:04.345476+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:33:04.345476+0000","last_undegraded":"2023-10-14T01:33:04.345476+0000","last_fullsized":"2023-10-14T01:33:04.345476+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'7697","reported_seq":10979,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.330536+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:33:04.330536+0000","last_peered":"2023-10-14T01:33:04.330536+0000","last_clean":"2023-10-14T01:33:04.330536+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:33:04.330536+0000","last_undegraded":"2023-10-14T01:33:04.330536+0000","last_fullsized":"2023-10-14T01:33:04.330536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'18967","reported_seq":27609,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.354956+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:33:04.354956+0000","last_peered":"2023-10-14T01:33:04.354956+0000","last_clean":"2023-10-14T01:33:04.354956+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:33:04.354956+0000","last_undegraded":"2023-10-14T01:33:04.354956+0000","last_fullsized":"2023-10-14T01:33:04.354956+0000","mapping_epoch":12,"log_start":"112'8900","ondisk_log_start":"112'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'601540","reported_seq":1196995,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.408730+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:33:04.408730+0000","last_peered":"2023-10-14T01:33:04.408730+0000","last_clean":"2023-10-14T01:33:04.408730+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:33:04.408730+0000","last_undegraded":"2023-10-14T01:33:04.408730+0000","last_fullsized":"2023-10-14T01:33:04.408730+0000","mapping_epoch":15,"log_start":"135'591536","ondisk_log_start":"135'591536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7438299136,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":597354,"num_read_kb":2356852,"num_write":596376,"num_write_kb":2385504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"136'3988","reported_seq":5823,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:03.855037+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:33:03.855037+0000","last_peered":"2023-10-14T01:33:03.855037+0000","last_clean":"2023-10-14T01:33:03.855037+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:33:03.855037+0000","last_undegraded":"2023-10-14T01:33:03.855037+0000","last_fullsized":"2023-10-14T01:33:03.855037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'570962","reported_seq":1136033,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:33:04.287101+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:33:04.287101+0000","last_peered":"2023-10-14T01:33:04.287101+0000","last_clean":"2023-10-14T01:33:04.287101+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:33:04.287101+0000","last_undegraded":"2023-10-14T01:33:04.287101+0000","last_fullsized":"2023-10-14T01:33:04.287101+0000","mapping_epoch":15,"log_start":"135'560928","ondisk_log_start":"135'560928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3061,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 684s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7065096192,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":566685,"num_read_kb":2235292,"num_write":566460,"num_write_kb":2265840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3331,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:03.868349+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:33:03.868349+0000","last_peered":"2023-10-14T01:33:03.868349+0000","last_clean":"2023-10-14T01:33:03.868349+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:33:03.868349+0000","last_undegraded":"2023-10-14T01:33:03.868349+0000","last_fullsized":"2023-10-14T01:33:03.868349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'593036","reported_seq":1180333,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.395722+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:33:04.395722+0000","last_peered":"2023-10-14T01:33:04.395722+0000","last_clean":"2023-10-14T01:33:04.395722+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:33:04.395722+0000","last_undegraded":"2023-10-14T01:33:04.395722+0000","last_fullsized":"2023-10-14T01:33:04.395722+0000","mapping_epoch":15,"log_start":"135'583028","ondisk_log_start":"135'583028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7321186304,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":589150,"num_read_kb":2324272,"num_write":587974,"num_write_kb":2351896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3481,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:03.855788+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:33:03.855788+0000","last_peered":"2023-10-14T01:33:03.855788+0000","last_clean":"2023-10-14T01:33:03.855788+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:33:03.855788+0000","last_undegraded":"2023-10-14T01:33:03.855788+0000","last_fullsized":"2023-10-14T01:33:03.855788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'564220","reported_seq":1122517,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:04.354031+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:33:04.354031+0000","last_peered":"2023-10-14T01:33:04.354031+0000","last_clean":"2023-10-14T01:33:04.354031+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:33:04.354031+0000","last_undegraded":"2023-10-14T01:33:04.354031+0000","last_fullsized":"2023-10-14T01:33:04.354031+0000","mapping_epoch":15,"log_start":"135'554127","ondisk_log_start":"135'554127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6966464531,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":560065,"num_read_kb":2208744,"num_write":559396,"num_write_kb":2237578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'11430","reported_seq":16237,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:33:03.858733+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:33:03.858733+0000","last_peered":"2023-10-14T01:33:03.858733+0000","last_clean":"2023-10-14T01:33:03.858733+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:33:03.858733+0000","last_undegraded":"2023-10-14T01:33:03.858733+0000","last_fullsized":"2023-10-14T01:33:03.858733+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":317,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:31:43.855112+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:31:43.855112+0000","last_peered":"2023-10-14T01:31:43.855112+0000","last_clean":"2023-10-14T01:31:43.855112+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:31:43.855112+0000","last_undegraded":"2023-10-14T01:31:43.855112+0000","last_fullsized":"2023-10-14T01:31:43.855112+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28791046163,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2313254,"num_read_kb":9125160,"num_write":2310206,"num_write_kb":9240818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18930278400,"data_stored":28112642086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40139,"ondisk_log_size":40139,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49815,"ondisk_log_size":49815,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739426,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10420244,"kb_used_data":9246856,"kb_used_omap":1482,"kb_used_meta":1171893,"kb_avail":83951596,"statfs":{"total":96636764160,"available":85966434304,"internally_reserved":0,"allocated":9468780544,"data_stored":14059084359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1200019427},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:06 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.40100000000000002},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.4900000000000002,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.186},{"interface":"front","average":{"1min":0.378,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.8600000000000001,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.28899999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:32:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.38400000000000001,"15min":0.50600000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52700000000000002,"5min":4.8150000000000004,"15min":15.845000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48099999999999998,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.81399999999999995,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739424,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6228564,"kb_used_data":4505548,"kb_used_omap":625,"kb_used_meta":1722382,"kb_avail":88143276,"statfs":{"total":96636764160,"available":90258714624,"internally_reserved":0,"allocated":4613681152,"data_stored":6854063952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1763720108},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,5,8,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38800000000000001,"15min":0.57699999999999996},"min":{"1min":0.22900000000000001,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":0.48599999999999999,"5min":3.399,"15min":24.248999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.374,"5min":0.442,"15min":0.56299999999999994},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.192},"max":{"1min":0.53500000000000003,"5min":3.488,"15min":17.263999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.41499999999999998,"15min":0.44900000000000001},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.50700000000000001,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.48799999999999999,"15min":0.57199999999999995},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.436,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.36199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739424,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5588012,"kb_used_data":4739300,"kb_used_omap":747,"kb_used_meta":847956,"kb_avail":88783828,"statfs":{"total":96636764160,"available":90914639872,"internally_reserved":0,"allocated":4853043200,"data_stored":7201828589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":868307024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,4,8,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.432,"15min":0.48899999999999999},"min":{"1min":0.18099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.56299999999999994,"5min":5.2359999999999998,"15min":16.023},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.44600000000000001,"15min":0.48899999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.17899999999999999},"max":{"1min":4.6360000000000001,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.439,"15min":0.46300000000000002},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.8069999999999999,"5min":5.141,"15min":5.141},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.51300000000000001,"15min":0.434},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":4.673,"5min":5.0609999999999999,"15min":5.0609999999999999},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4851097600,"data_stored":7200825344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4611506176,"data_stored":6852960275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9467674624,"data_stored":14058856467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:33:08.022 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:33:08.023 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:33:28.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:33:28.661 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:33:28.662 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:33:28.704 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2784,"stamp":"2023-10-14T01:33:27.682033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28824669925,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2355311,"num_read_kb":9187114,"num_write":2319487,"num_write_kb":9277007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90118,"ondisk_log_size":90118,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22490608,"kb_used_data":18558744,"kb_used_omap":2855,"kb_used_meta":3928984,"kb_avail":260624912,"statfs":{"total":289910292480,"available":266879909888,"internally_reserved":0,"allocated":19004153856,"data_stored":28183617402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2923553,"internal_metadata":4023280607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,14,14,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16822272,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4289,"num_read_kb":17116,"num_write":4324,"num_write_kb":17296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001952"},"pg_stats":[{"pgid":"2.7","version":"137'7738","reported_seq":11230,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.335434+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:33:24.335434+0000","last_peered":"2023-10-14T01:33:24.335434+0000","last_clean":"2023-10-14T01:33:24.335434+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:33:24.335434+0000","last_undegraded":"2023-10-14T01:33:24.335434+0000","last_fullsized":"2023-10-14T01:33:24.335434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'11533","reported_seq":17940,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.335415+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:33:24.335415+0000","last_peered":"2023-10-14T01:33:24.335415+0000","last_clean":"2023-10-14T01:33:24.335415+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:33:24.335415+0000","last_undegraded":"2023-10-14T01:33:24.335415+0000","last_fullsized":"2023-10-14T01:33:24.335415+0000","mapping_epoch":12,"log_start":"55'1500","ondisk_log_start":"55'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'7722","reported_seq":11014,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.335289+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:33:24.335289+0000","last_peered":"2023-10-14T01:33:24.335289+0000","last_clean":"2023-10-14T01:33:24.335289+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:33:24.335289+0000","last_undegraded":"2023-10-14T01:33:24.335289+0000","last_fullsized":"2023-10-14T01:33:24.335289+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'19028","reported_seq":27693,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.339016+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:33:24.339016+0000","last_peered":"2023-10-14T01:33:24.339016+0000","last_clean":"2023-10-14T01:33:24.339016+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:33:24.339016+0000","last_undegraded":"2023-10-14T01:33:24.339016+0000","last_fullsized":"2023-10-14T01:33:24.339016+0000","mapping_epoch":12,"log_start":"114'9000","ondisk_log_start":"114'9000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'603731","reported_seq":1201277,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.578060+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:33:24.578060+0000","last_peered":"2023-10-14T01:33:24.578060+0000","last_clean":"2023-10-14T01:33:24.578060+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:33:24.578060+0000","last_undegraded":"2023-10-14T01:33:24.578060+0000","last_fullsized":"2023-10-14T01:33:24.578060+0000","mapping_epoch":15,"log_start":"136'593636","ondisk_log_start":"136'593636","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7446896640,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":599446,"num_read_kb":2365188,"num_write":598567,"num_write_kb":2394268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"137'4000","reported_seq":5841,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:23.536281+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:33:23.536281+0000","last_peered":"2023-10-14T01:33:23.536281+0000","last_clean":"2023-10-14T01:33:23.536281+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:33:23.536281+0000","last_undegraded":"2023-10-14T01:33:23.536281+0000","last_fullsized":"2023-10-14T01:33:23.536281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4000,"log_dups_size":0,"ondisk_log_size":4000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'573083","reported_seq":1140221,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:33:24.505321+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:33:24.505321+0000","last_peered":"2023-10-14T01:33:24.505321+0000","last_clean":"2023-10-14T01:33:24.505321+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:33:24.505321+0000","last_undegraded":"2023-10-14T01:33:24.505321+0000","last_fullsized":"2023-10-14T01:33:24.505321+0000","mapping_epoch":15,"log_start":"136'563028","ondisk_log_start":"136'563028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3178,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 704s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7073042432,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":568740,"num_read_kb":2243496,"num_write":568501,"num_write_kb":2274004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3339,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.335032+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:33:24.335032+0000","last_peered":"2023-10-14T01:33:24.335032+0000","last_clean":"2023-10-14T01:33:24.335032+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:33:24.335032+0000","last_undegraded":"2023-10-14T01:33:24.335032+0000","last_fullsized":"2023-10-14T01:33:24.335032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'595231","reported_seq":1184672,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.568023+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:33:24.568023+0000","last_peered":"2023-10-14T01:33:24.568023+0000","last_clean":"2023-10-14T01:33:24.568023+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:33:24.568023+0000","last_undegraded":"2023-10-14T01:33:24.568023+0000","last_fullsized":"2023-10-14T01:33:24.568023+0000","mapping_epoch":15,"log_start":"136'585228","ondisk_log_start":"136'585228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7329660928,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":591288,"num_read_kb":2332800,"num_write":590169,"num_write_kb":2360676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3489,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.340446+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:33:24.340446+0000","last_peered":"2023-10-14T01:33:24.340446+0000","last_clean":"2023-10-14T01:33:24.340446+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:33:24.340446+0000","last_undegraded":"2023-10-14T01:33:24.340446+0000","last_fullsized":"2023-10-14T01:33:24.340446+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'566278","reported_seq":1126601,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.506180+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:33:24.506180+0000","last_peered":"2023-10-14T01:33:24.506180+0000","last_clean":"2023-10-14T01:33:24.506180+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:33:24.506180+0000","last_undegraded":"2023-10-14T01:33:24.506180+0000","last_fullsized":"2023-10-14T01:33:24.506180+0000","mapping_epoch":15,"log_start":"135'556227","ondisk_log_start":"135'556227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6974476307,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":562089,"num_read_kb":2216824,"num_write":561454,"num_write_kb":2245810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'11470","reported_seq":16291,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:24.339384+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:33:24.339384+0000","last_peered":"2023-10-14T01:33:24.339384+0000","last_clean":"2023-10-14T01:33:24.339384+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:33:24.339384+0000","last_undegraded":"2023-10-14T01:33:24.339384+0000","last_fullsized":"2023-10-14T01:33:24.339384+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":318,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:31:46.550750+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:31:46.550750+0000","last_peered":"2023-10-14T01:31:46.550750+0000","last_clean":"2023-10-14T01:31:46.550750+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:31:46.550750+0000","last_undegraded":"2023-10-14T01:31:46.550750+0000","last_fullsized":"2023-10-14T01:31:46.550750+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28824076307,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2321563,"num_read_kb":9158308,"num_write":2318691,"num_write_kb":9274758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18999189504,"data_stored":28181553190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49815,"ondisk_log_size":49815,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739430,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10561880,"kb_used_data":9280272,"kb_used_omap":1482,"kb_used_meta":1280117,"kb_avail":83809960,"statfs":{"total":96636764160,"available":85821399040,"internally_reserved":0,"allocated":9502998528,"data_stored":14093302343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1310840803},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:06 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.40100000000000002},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.4900000000000002,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.8600000000000001,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.498}]},{"osd":1,"last update":"Sat Oct 14 01:32:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.38400000000000001,"15min":0.50600000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52700000000000002,"5min":4.8150000000000004,"15min":15.845000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48099999999999998,"15min":0.57699999999999996},"min":{"1min":0.214,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.81399999999999995,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.73999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739428,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6298820,"kb_used_data":4521660,"kb_used_omap":625,"kb_used_meta":1776526,"kb_avail":88073020,"statfs":{"total":96636764160,"available":90186772480,"internally_reserved":0,"allocated":4630179840,"data_stored":6870558411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1819163564},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,7,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.38800000000000001,"15min":0.57699999999999996},"min":{"1min":0.22900000000000001,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":0.48599999999999999,"5min":3.399,"15min":24.248999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.442,"15min":0.56299999999999994},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.192},"max":{"1min":0.53500000000000003,"5min":3.488,"15min":17.263999999999999},"last":0.44500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:32:22 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.41499999999999998,"15min":0.44900000000000001},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.50700000000000001,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.313},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.48799999999999999,"15min":0.57199999999999995},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.436,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5629908,"kb_used_data":4756812,"kb_used_omap":747,"kb_used_meta":872340,"kb_avail":88741932,"statfs":{"total":96636764160,"available":90871738368,"internally_reserved":0,"allocated":4870975488,"data_stored":7219756648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":893276240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.52500000000000002,"15min":0.51200000000000001},"min":{"1min":0.215,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":16.023},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.48699999999999999},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":1.4179999999999999,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40000000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.5129999999999999,"5min":2.8069999999999999,"15min":5.141},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.44500000000000001,"5min":4.673,"15min":5.0609999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4869070848,"data_stored":7218798592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4628295680,"data_stored":6869749779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9501822976,"data_stored":14093004819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:33:28.706 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:33:28.707 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:33:38.038 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:38.036+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:33:38.043 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:38.040+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:33:38.044 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:38.040+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:33:38.060 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:38.056+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:33:38.061 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:33:38.056+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:33:48.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:33:49.163 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:33:49.165 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:33:49.195 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2794,"stamp":"2023-10-14T01:33:47.687955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28856684261,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2363569,"num_read_kb":9220070,"num_write":2327749,"num_write_kb":9310055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90139,"ondisk_log_size":90139,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22765240,"kb_used_data":18625244,"kb_used_omap":2855,"kb_used_meta":4137112,"kb_avail":260350280,"statfs":{"total":289910292480,"available":266598686720,"internally_reserved":0,"allocated":19072249856,"data_stored":28251713402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2923553,"internal_metadata":4236403679},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,6,8,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16191488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4240,"num_read_kb":16924,"num_write":4180,"num_write_kb":16720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004554"},"pg_stats":[{"pgid":"2.7","version":"137'7771","reported_seq":11269,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:43.219129+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:33:43.219129+0000","last_peered":"2023-10-14T01:33:43.219129+0000","last_clean":"2023-10-14T01:33:43.219129+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:33:43.219129+0000","last_undegraded":"2023-10-14T01:33:43.219129+0000","last_fullsized":"2023-10-14T01:33:43.219129+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'11582","reported_seq":17998,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:43.232534+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:33:43.232534+0000","last_peered":"2023-10-14T01:33:43.232534+0000","last_clean":"2023-10-14T01:33:43.232534+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:33:43.232534+0000","last_undegraded":"2023-10-14T01:33:43.232534+0000","last_fullsized":"2023-10-14T01:33:43.232534+0000","mapping_epoch":12,"log_start":"55'1500","ondisk_log_start":"55'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'7756","reported_seq":11054,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:43.199080+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:33:43.199080+0000","last_peered":"2023-10-14T01:33:43.199080+0000","last_clean":"2023-10-14T01:33:43.199080+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:33:43.199080+0000","last_undegraded":"2023-10-14T01:33:43.199080+0000","last_fullsized":"2023-10-14T01:33:43.199080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'19109","reported_seq":27789,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:43.229599+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:33:43.229599+0000","last_peered":"2023-10-14T01:33:43.229599+0000","last_clean":"2023-10-14T01:33:43.229599+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:33:43.229599+0000","last_undegraded":"2023-10-14T01:33:43.229599+0000","last_fullsized":"2023-10-14T01:33:43.229599+0000","mapping_epoch":12,"log_start":"114'9100","ondisk_log_start":"114'9100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'605840","reported_seq":1205548,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.610359+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:33:44.610359+0000","last_peered":"2023-10-14T01:33:44.610359+0000","last_clean":"2023-10-14T01:33:44.610359+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:33:44.610359+0000","last_undegraded":"2023-10-14T01:33:44.610359+0000","last_fullsized":"2023-10-14T01:33:44.610359+0000","mapping_epoch":15,"log_start":"136'595836","ondisk_log_start":"136'595836","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7455121408,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":601604,"num_read_kb":2373792,"num_write":600676,"num_write_kb":2402704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"137'4020","reported_seq":5864,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:43.654216+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:33:43.654216+0000","last_peered":"2023-10-14T01:33:43.654216+0000","last_clean":"2023-10-14T01:33:43.654216+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:33:43.654216+0000","last_undegraded":"2023-10-14T01:33:43.654216+0000","last_fullsized":"2023-10-14T01:33:43.654216+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4020,"log_dups_size":0,"ondisk_log_size":4020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'575233","reported_seq":1144409,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:33:44.612081+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:33:44.612081+0000","last_peered":"2023-10-14T01:33:44.612081+0000","last_clean":"2023-10-14T01:33:44.612081+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:33:44.612081+0000","last_undegraded":"2023-10-14T01:33:44.612081+0000","last_fullsized":"2023-10-14T01:33:44.612081+0000","mapping_epoch":15,"log_start":"136'565228","ondisk_log_start":"136'565228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3314,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 724s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7081037824,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":570768,"num_read_kb":2251600,"num_write":570558,"num_write_kb":2282232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3347,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:45.011218+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:33:45.011218+0000","last_peered":"2023-10-14T01:33:45.011218+0000","last_clean":"2023-10-14T01:33:45.011218+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:33:45.011218+0000","last_undegraded":"2023-10-14T01:33:45.011218+0000","last_fullsized":"2023-10-14T01:33:45.011218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'597318","reported_seq":1188860,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.610513+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:33:44.610513+0000","last_peered":"2023-10-14T01:33:44.610513+0000","last_clean":"2023-10-14T01:33:44.610513+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:33:44.610513+0000","last_undegraded":"2023-10-14T01:33:44.610513+0000","last_fullsized":"2023-10-14T01:33:44.610513+0000","mapping_epoch":15,"log_start":"136'587228","ondisk_log_start":"136'587228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7337705472,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":593395,"num_read_kb":2341216,"num_write":592256,"num_write_kb":2369024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3498,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.862682+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:33:44.862682+0000","last_peered":"2023-10-14T01:33:44.862682+0000","last_clean":"2023-10-14T01:33:44.862682+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:33:44.862682+0000","last_undegraded":"2023-10-14T01:33:44.862682+0000","last_fullsized":"2023-10-14T01:33:44.862682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'568287","reported_seq":1130574,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.599159+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:33:44.599159+0000","last_peered":"2023-10-14T01:33:44.599159+0000","last_clean":"2023-10-14T01:33:44.599159+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:33:44.599159+0000","last_undegraded":"2023-10-14T01:33:44.599159+0000","last_fullsized":"2023-10-14T01:33:44.599159+0000","mapping_epoch":15,"log_start":"136'558227","ondisk_log_start":"136'558227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6982225939,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":564054,"num_read_kb":2224656,"num_write":563463,"num_write_kb":2253846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'11519","reported_seq":16349,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:43.208081+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:33:43.208081+0000","last_peered":"2023-10-14T01:33:43.208081+0000","last_clean":"2023-10-14T01:33:43.208081+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:33:43.208081+0000","last_undegraded":"2023-10-14T01:33:43.208081+0000","last_fullsized":"2023-10-14T01:33:43.208081+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":319,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:06.566591+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:33:06.566591+0000","last_peered":"2023-10-14T01:33:06.566591+0000","last_clean":"2023-10-14T01:33:06.566591+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:33:06.566591+0000","last_undegraded":"2023-10-14T01:33:06.566591+0000","last_fullsized":"2023-10-14T01:33:06.566591+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28856090643,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2329821,"num_read_kb":9191264,"num_write":2326953,"num_write_kb":9307806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19067559936,"data_stored":28249923622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40159,"ondisk_log_size":40159,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49881,"ondisk_log_size":49881,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739434,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10695356,"kb_used_data":9313904,"kb_used_omap":1482,"kb_used_meta":1379957,"kb_avail":83676484,"statfs":{"total":96636764160,"available":85684719616,"internally_reserved":0,"allocated":9537437696,"data_stored":14127741511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1413076963},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:06 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.40100000000000002},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.4900000000000002,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.8600000000000001,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.30399999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:33:34 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.33400000000000002,"15min":0.49199999999999999},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.40600000000000003,"5min":0.82199999999999995,"15min":15.845000000000001},"last":0.251},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.874,"5min":4.8419999999999996,"15min":15.805999999999999},"last":2.2629999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739432,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6369124,"kb_used_data":4537820,"kb_used_omap":625,"kb_used_meta":1830670,"kb_avail":88002716,"statfs":{"total":96636764160,"available":90114781184,"internally_reserved":0,"allocated":4646727680,"data_stored":6887106251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1874607020},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,5,13,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:25 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.57999999999999996},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":1.9690000000000001,"5min":3.399,"15min":24.248999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.44400000000000001,"15min":0.56299999999999994},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.192},"max":{"1min":2.0369999999999999,"5min":3.488,"15min":17.263999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Sat Oct 14 01:33:25 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.41499999999999998,"15min":0.44400000000000001},"min":{"1min":0.20499999999999999,"5min":0.193,"15min":0.185},"max":{"1min":0.52200000000000002,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.311,"5min":0.47199999999999998,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5700760,"kb_used_data":4773520,"kb_used_omap":747,"kb_used_meta":926484,"kb_avail":88671080,"statfs":{"total":96636764160,"available":90799185920,"internally_reserved":0,"allocated":4888084480,"data_stored":7236865640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":948719696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.52500000000000002,"15min":0.51200000000000001},"min":{"1min":0.215,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":16.023},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.48699999999999999},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":1.4179999999999999,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.23499999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40000000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.5129999999999999,"5min":2.8069999999999999,"15min":5.141},"last":0.499},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.44500000000000001,"5min":4.673,"15min":5.0609999999999999},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4886253568,"data_stored":7235981312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4644950016,"data_stored":6886404115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9536356352,"data_stored":14127538195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:33:49.197 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:33:49.197 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:34:09.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:34:09.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:34:09.969 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:34:10.020 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2805,"stamp":"2023-10-14T01:34:09.691567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28889800421,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2372097,"num_read_kb":9254110,"num_write":2336312,"num_write_kb":9344307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90302,"ondisk_log_size":90302,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23082760,"kb_used_data":18693612,"kb_used_omap":2855,"kb_used_meta":4386264,"kb_avail":260032760,"statfs":{"total":289910292480,"available":266273546240,"internally_reserved":0,"allocated":19142258688,"data_stored":28321722234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2923553,"internal_metadata":4491535327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,14,5,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16568320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4283,"num_read_kb":17076,"num_write":4275,"num_write_kb":17100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001939"},"pg_stats":[{"pgid":"2.7","version":"137'7789","reported_seq":11295,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:02.989914+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:34:02.989914+0000","last_peered":"2023-10-14T01:34:02.989914+0000","last_clean":"2023-10-14T01:34:02.989914+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:34:02.989914+0000","last_undegraded":"2023-10-14T01:34:02.989914+0000","last_fullsized":"2023-10-14T01:34:02.989914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'11609","reported_seq":18037,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:02.999990+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:34:02.999990+0000","last_peered":"2023-10-14T01:34:02.999990+0000","last_clean":"2023-10-14T01:34:02.999990+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:34:02.999990+0000","last_undegraded":"2023-10-14T01:34:02.999990+0000","last_fullsized":"2023-10-14T01:34:02.999990+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'7774","reported_seq":11080,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:02.980626+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:34:02.980626+0000","last_peered":"2023-10-14T01:34:02.980626+0000","last_clean":"2023-10-14T01:34:02.980626+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:34:02.980626+0000","last_undegraded":"2023-10-14T01:34:02.980626+0000","last_fullsized":"2023-10-14T01:34:02.980626+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'19154","reported_seq":27854,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:03.022140+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:34:03.022140+0000","last_peered":"2023-10-14T01:34:03.022140+0000","last_clean":"2023-10-14T01:34:03.022140+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:34:03.022140+0000","last_undegraded":"2023-10-14T01:34:03.022140+0000","last_fullsized":"2023-10-14T01:34:03.022140+0000","mapping_epoch":12,"log_start":"114'9100","ondisk_log_start":"114'9100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'608094","reported_seq":1210054,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:04.681850+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:34:04.681850+0000","last_peered":"2023-10-14T01:34:04.681850+0000","last_clean":"2023-10-14T01:34:04.681850+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:34:04.681850+0000","last_undegraded":"2023-10-14T01:34:04.681850+0000","last_fullsized":"2023-10-14T01:34:04.681850+0000","mapping_epoch":15,"log_start":"136'598036","ondisk_log_start":"136'598036","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7463915520,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":603854,"num_read_kb":2382780,"num_write":602930,"num_write_kb":2411720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"137'4041","reported_seq":5889,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:02.458602+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:34:02.458602+0000","last_peered":"2023-10-14T01:34:02.458602+0000","last_clean":"2023-10-14T01:34:02.458602+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:34:02.458602+0000","last_undegraded":"2023-10-14T01:34:02.458602+0000","last_fullsized":"2023-10-14T01:34:02.458602+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'577312","reported_seq":1148511,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:34:04.695244+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:34:04.695244+0000","last_peered":"2023-10-14T01:34:04.695244+0000","last_clean":"2023-10-14T01:34:04.695244+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:34:04.695244+0000","last_undegraded":"2023-10-14T01:34:04.695244+0000","last_fullsized":"2023-10-14T01:34:04.695244+0000","mapping_epoch":15,"log_start":"136'567229","ondisk_log_start":"136'567229","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3377,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 745s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7088988160,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":572782,"num_read_kb":2259628,"num_write":572592,"num_write_kb":2290368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3355,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:05.304814+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:34:05.304814+0000","last_peered":"2023-10-14T01:34:05.304814+0000","last_clean":"2023-10-14T01:34:05.304814+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:34:05.304814+0000","last_undegraded":"2023-10-14T01:34:05.304814+0000","last_fullsized":"2023-10-14T01:34:05.304814+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'599499","reported_seq":1193237,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:04.681349+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:34:04.681349+0000","last_peered":"2023-10-14T01:34:04.681349+0000","last_clean":"2023-10-14T01:34:04.681349+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:34:04.681349+0000","last_undegraded":"2023-10-14T01:34:04.681349+0000","last_fullsized":"2023-10-14T01:34:04.681349+0000","mapping_epoch":15,"log_start":"136'589428","ondisk_log_start":"136'589428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7346003968,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":595585,"num_read_kb":2349960,"num_write":594437,"num_write_kb":2377748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3506,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:05.300524+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:34:05.300524+0000","last_peered":"2023-10-14T01:34:05.300524+0000","last_clean":"2023-10-14T01:34:05.300524+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:34:05.300524+0000","last_undegraded":"2023-10-14T01:34:05.300524+0000","last_fullsized":"2023-10-14T01:34:05.300524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'570381","reported_seq":1134741,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:04.722593+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:34:04.722593+0000","last_peered":"2023-10-14T01:34:04.722593+0000","last_clean":"2023-10-14T01:34:04.722593+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:34:04.722593+0000","last_undegraded":"2023-10-14T01:34:04.722593+0000","last_fullsized":"2023-10-14T01:34:04.722593+0000","mapping_epoch":15,"log_start":"136'560327","ondisk_log_start":"136'560327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6990299155,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":566128,"num_read_kb":2232936,"num_write":565557,"num_write_kb":2262222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'11546","reported_seq":16388,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:34:03.040566+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:34:03.040566+0000","last_peered":"2023-10-14T01:34:03.040566+0000","last_clean":"2023-10-14T01:34:03.040566+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:34:03.040566+0000","last_undegraded":"2023-10-14T01:34:03.040566+0000","last_fullsized":"2023-10-14T01:34:03.040566+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":319,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:06.566591+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:33:06.566591+0000","last_peered":"2023-10-14T01:33:06.566591+0000","last_clean":"2023-10-14T01:33:06.566591+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:33:06.566591+0000","last_undegraded":"2023-10-14T01:33:06.566591+0000","last_fullsized":"2023-10-14T01:33:06.566591+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28889206803,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2338349,"num_read_kb":9225304,"num_write":2335516,"num_write_kb":9342058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19137536000,"data_stored":28319899686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40266,"ondisk_log_size":40266,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":49937,"ondisk_log_size":49937,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739438,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10837864,"kb_used_data":9348188,"kb_used_omap":1482,"kb_used_meta":1488181,"kb_avail":83533976,"statfs":{"total":96636764160,"available":85538791424,"internally_reserved":0,"allocated":9572544512,"data_stored":14162848327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1523898339},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:06 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.40100000000000002},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":3.4900000000000002,"5min":4.8760000000000003,"15min":8.8360000000000003},"last":0.254},{"interface":"front","average":{"1min":0.378,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":1.8600000000000001,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.34599999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:33:34 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.33400000000000002,"15min":0.49199999999999999},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.40600000000000003,"5min":0.82199999999999995,"15min":15.845000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.874,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.38900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739436,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6450160,"kb_used_data":4554280,"kb_used_omap":625,"kb_used_meta":1895246,"kb_avail":87921680,"statfs":{"total":96636764160,"available":90031800320,"internally_reserved":0,"allocated":4663582720,"data_stored":6903961291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1940732844},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:25 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.57999999999999996},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":1.9690000000000001,"5min":3.399,"15min":24.248999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.435,"5min":0.44400000000000001,"15min":0.56299999999999994},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.192},"max":{"1min":2.0369999999999999,"5min":3.488,"15min":17.263999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Oct 14 01:33:25 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.41499999999999998,"15min":0.44400000000000001},"min":{"1min":0.20499999999999999,"5min":0.193,"15min":0.185},"max":{"1min":0.52200000000000002,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.311,"5min":0.47199999999999998,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359739436,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5794736,"kb_used_data":4791144,"kb_used_omap":747,"kb_used_meta":1002836,"kb_avail":88577104,"statfs":{"total":96636764160,"available":90702954496,"internally_reserved":0,"allocated":4906131456,"data_stored":7254912616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":1026904144},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.52500000000000002,"15min":0.51200000000000001},"min":{"1min":0.215,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":16.023},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.48699999999999999},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":1.4179999999999999,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40000000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.5129999999999999,"5min":2.8069999999999999,"15min":5.141},"last":0.245},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.44500000000000001,"5min":4.673,"15min":5.0609999999999999},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4904415232,"data_stored":7254142976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4661866496,"data_stored":6903320595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9571254272,"data_stored":14162436115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:34:10.022 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:34:10.023 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:34:16.119 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:34:16.107+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:34:16.136 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:34:16.127+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:34:25.793 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:34:25.787+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:34:25.794 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:34:25.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:34:25.794 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:34:25.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:34:25.795 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:34:25.791+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:34:30.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:34:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:34:30.554 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:34:30.599 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2816,"stamp":"2023-10-14T01:34:29.697417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28922281701,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2380536,"num_read_kb":9287750,"num_write":2344702,"num_write_kb":9377867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90363,"ondisk_log_size":90363,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23375548,"kb_used_data":18760164,"kb_used_omap":2855,"kb_used_meta":4612504,"kb_avail":259739972,"statfs":{"total":289910292480,"available":265973731328,"internally_reserved":0,"allocated":19210407936,"data_stored":28389858666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2923553,"internal_metadata":4723205087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,4,5,14,21,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16252928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4205,"num_read_kb":16760,"num_write":4205,"num_write_kb":16820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004508"},"pg_stats":[{"pgid":"2.7","version":"138'7817","reported_seq":11332,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:23.595089+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:34:23.595089+0000","last_peered":"2023-10-14T01:34:23.595089+0000","last_clean":"2023-10-14T01:34:23.595089+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:34:23.595089+0000","last_undegraded":"2023-10-14T01:34:23.595089+0000","last_fullsized":"2023-10-14T01:34:23.595089+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7817,"log_dups_size":0,"ondisk_log_size":7817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'11651","reported_seq":18093,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:23.605913+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:34:23.605913+0000","last_peered":"2023-10-14T01:34:23.605913+0000","last_clean":"2023-10-14T01:34:23.605913+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:34:23.605913+0000","last_undegraded":"2023-10-14T01:34:23.605913+0000","last_fullsized":"2023-10-14T01:34:23.605913+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'7803","reported_seq":11118,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:23.595389+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:34:23.595389+0000","last_peered":"2023-10-14T01:34:23.595389+0000","last_clean":"2023-10-14T01:34:23.595389+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:34:23.595389+0000","last_undegraded":"2023-10-14T01:34:23.595389+0000","last_fullsized":"2023-10-14T01:34:23.595389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7803,"log_dups_size":0,"ondisk_log_size":7803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'19225","reported_seq":27946,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:23.612603+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:34:23.612603+0000","last_peered":"2023-10-14T01:34:23.612603+0000","last_clean":"2023-10-14T01:34:23.612603+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:34:23.612603+0000","last_undegraded":"2023-10-14T01:34:23.612603+0000","last_fullsized":"2023-10-14T01:34:23.612603+0000","mapping_epoch":12,"log_start":"114'9200","ondisk_log_start":"114'9200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'610282","reported_seq":1214455,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:24.773502+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:34:24.773502+0000","last_peered":"2023-10-14T01:34:24.773502+0000","last_clean":"2023-10-14T01:34:24.773502+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:34:24.773502+0000","last_undegraded":"2023-10-14T01:34:24.773502+0000","last_fullsized":"2023-10-14T01:34:24.773502+0000","mapping_epoch":15,"log_start":"136'600236","ondisk_log_start":"136'600236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7472414720,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":606069,"num_read_kb":2391588,"num_write":605118,"num_write_kb":2420472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'4055","reported_seq":5908,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:23.449325+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:34:23.449325+0000","last_peered":"2023-10-14T01:34:23.449325+0000","last_clean":"2023-10-14T01:34:23.449325+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:34:23.449325+0000","last_undegraded":"2023-10-14T01:34:23.449325+0000","last_fullsized":"2023-10-14T01:34:23.449325+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'579405","reported_seq":1152687,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:34:24.840997+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:34:24.840997+0000","last_peered":"2023-10-14T01:34:24.840997+0000","last_clean":"2023-10-14T01:34:24.840997+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:34:24.840997+0000","last_undegraded":"2023-10-14T01:34:24.840997+0000","last_fullsized":"2023-10-14T01:34:24.840997+0000","mapping_epoch":15,"log_start":"136'569329","ondisk_log_start":"136'569329","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3443,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7096909824,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":574849,"num_read_kb":2267868,"num_write":574640,"num_write_kb":2298560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3364,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:25.779007+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:34:25.779007+0000","last_peered":"2023-10-14T01:34:25.779007+0000","last_clean":"2023-10-14T01:34:25.779007+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:34:25.779007+0000","last_undegraded":"2023-10-14T01:34:25.779007+0000","last_fullsized":"2023-10-14T01:34:25.779007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'601667","reported_seq":1197491,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:24.783103+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:34:24.783103+0000","last_peered":"2023-10-14T01:34:24.783103+0000","last_clean":"2023-10-14T01:34:24.783103+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:34:24.783103+0000","last_undegraded":"2023-10-14T01:34:24.783103+0000","last_fullsized":"2023-10-14T01:34:24.783103+0000","mapping_epoch":15,"log_start":"136'591628","ondisk_log_start":"136'591628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7354343424,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":597671,"num_read_kb":2358288,"num_write":596605,"num_write_kb":2386420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3515,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:25.784554+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:34:25.784554+0000","last_peered":"2023-10-14T01:34:25.784554+0000","last_clean":"2023-10-14T01:34:25.784554+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:34:25.784554+0000","last_undegraded":"2023-10-14T01:34:25.784554+0000","last_fullsized":"2023-10-14T01:34:25.784554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'572367","reported_seq":1138802,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:24.855044+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:34:24.855044+0000","last_peered":"2023-10-14T01:34:24.855044+0000","last_clean":"2023-10-14T01:34:24.855044+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:34:24.855044+0000","last_undegraded":"2023-10-14T01:34:24.855044+0000","last_fullsized":"2023-10-14T01:34:24.855044+0000","mapping_epoch":15,"log_start":"136'562327","ondisk_log_start":"136'562327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":6998020115,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":568199,"num_read_kb":2241200,"num_write":567543,"num_write_kb":2270166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"138'11588","reported_seq":16443,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:23.622490+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:34:23.622490+0000","last_peered":"2023-10-14T01:34:23.622490+0000","last_clean":"2023-10-14T01:34:23.622490+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:34:23.622490+0000","last_undegraded":"2023-10-14T01:34:23.622490+0000","last_fullsized":"2023-10-14T01:34:23.622490+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":320,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.612203+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:33:44.612203+0000","last_peered":"2023-10-14T01:33:44.612203+0000","last_clean":"2023-10-14T01:33:44.612203+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:33:44.612203+0000","last_undegraded":"2023-10-14T01:33:44.612203+0000","last_fullsized":"2023-10-14T01:33:44.612203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28921688083,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2346788,"num_read_kb":9258944,"num_write":2343906,"num_write_kb":9375618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19206123520,"data_stored":28388487206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40201,"ondisk_log_size":40201,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":50063,"ondisk_log_size":50063,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739442,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10988880,"kb_used_data":9381256,"kb_used_omap":1482,"kb_used_meta":1606133,"kb_avail":83382960,"statfs":{"total":96636764160,"available":85384151040,"internally_reserved":0,"allocated":9606406144,"data_stored":14196705687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1644681187},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:34:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42099999999999999,"15min":0.40100000000000002},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.5740000000000001,"5min":4.1369999999999996,"15min":8.8360000000000003},"last":0.189},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.193},"max":{"1min":2.1989999999999998,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.23599999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:33:34 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.33400000000000002,"15min":0.49199999999999999},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.40600000000000003,"5min":0.82199999999999995,"15min":15.845000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.56999999999999995},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":0.874,"5min":4.8419999999999996,"15min":15.805999999999999},"last":0.254}]}]},{"osd":1,"up_from":8,"seq":34359739440,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6520504,"kb_used_data":4570480,"kb_used_omap":625,"kb_used_meta":1949390,"kb_avail":87851336,"statfs":{"total":96636764160,"available":89959768064,"internally_reserved":0,"allocated":4680171520,"data_stored":6920545819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":640084,"internal_metadata":1996176300},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,3,10,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:33:25 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.57999999999999996},"min":{"1min":0.17999999999999999,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":1.9690000000000001,"5min":3.399,"15min":24.248999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.44400000000000001,"15min":0.56299999999999994},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.192},"max":{"1min":2.0369999999999999,"5min":3.488,"15min":17.263999999999999},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:33:25 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.41499999999999998,"15min":0.44400000000000001},"min":{"1min":0.20499999999999999,"5min":0.193,"15min":0.185},"max":{"1min":0.52200000000000002,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.311,"5min":0.47199999999999998,"15min":0.56799999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":6.2800000000000002,"15min":22.498999999999999},"last":0.40100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739440,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5866164,"kb_used_data":4808428,"kb_used_omap":747,"kb_used_meta":1056980,"kb_avail":88505676,"statfs":{"total":96636764160,"available":90629812224,"internally_reserved":0,"allocated":4923830272,"data_stored":7272607160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":1082347600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,4,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.52700000000000002,"15min":0.48899999999999999},"min":{"1min":0.20499999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.65800000000000003,"5min":5.7549999999999999,"15min":16.023},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.22900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.624,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.33800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39500000000000002,"15min":0.45500000000000002},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.17699999999999999},"max":{"1min":1.5629999999999999,"5min":2.8069999999999999,"15min":5.141},"last":2.6110000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.43099999999999999,"15min":0.41899999999999998},"min":{"1min":0.22800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.80200000000000005,"5min":4.673,"15min":5.0609999999999999},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4922265600,"data_stored":7271993344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4678393856,"data_stored":6919847955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9605464064,"data_stored":14196645907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:34:30.601 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:34:30.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:34:31.161 INFO:teuthology.orchestra.run.smithi081.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:34:31.196 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:34:51.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:34:51.761 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:34:51.762 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:34:51.793 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2827,"stamp":"2023-10-14T01:34:51.701128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28962270949,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2390918,"num_read_kb":9329138,"num_write":2355052,"num_write_kb":9419267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90386,"ondisk_log_size":90386,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23310940,"kb_used_data":18834304,"kb_used_omap":2856,"kb_used_meta":4473751,"kb_avail":259804580,"statfs":{"total":289910292480,"available":266039889920,"internally_reserved":0,"allocated":19286327296,"data_stored":28465778026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2925232,"internal_metadata":4581121360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,6,9,16,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24215552,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6176,"num_read_kb":24604,"num_write":6262,"num_write_kb":25048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002080"},"pg_stats":[{"pgid":"2.7","version":"138'7866","reported_seq":11392,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:48.856908+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:34:48.856908+0000","last_peered":"2023-10-14T01:34:48.856908+0000","last_clean":"2023-10-14T01:34:48.856908+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:34:48.856908+0000","last_undegraded":"2023-10-14T01:34:48.856908+0000","last_fullsized":"2023-10-14T01:34:48.856908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'11724","reported_seq":18181,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:48.867123+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:34:48.867123+0000","last_peered":"2023-10-14T01:34:48.867123+0000","last_clean":"2023-10-14T01:34:48.867123+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:34:48.867123+0000","last_undegraded":"2023-10-14T01:34:48.867123+0000","last_fullsized":"2023-10-14T01:34:48.867123+0000","mapping_epoch":12,"log_start":"58'1700","ondisk_log_start":"58'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'7852","reported_seq":11177,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:48.867197+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:34:48.867197+0000","last_peered":"2023-10-14T01:34:48.867197+0000","last_clean":"2023-10-14T01:34:48.867197+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:34:48.867197+0000","last_undegraded":"2023-10-14T01:34:48.867197+0000","last_fullsized":"2023-10-14T01:34:48.867197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'19346","reported_seq":28092,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:48.875097+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:34:48.875097+0000","last_peered":"2023-10-14T01:34:48.875097+0000","last_clean":"2023-10-14T01:34:48.875097+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:34:48.875097+0000","last_undegraded":"2023-10-14T01:34:48.875097+0000","last_fullsized":"2023-10-14T01:34:48.875097+0000","mapping_epoch":12,"log_start":"114'9300","ondisk_log_start":"114'9300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'613049","reported_seq":1219855,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:49.952083+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:34:49.952083+0000","last_peered":"2023-10-14T01:34:49.952083+0000","last_clean":"2023-10-14T01:34:49.952083+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:34:49.952083+0000","last_undegraded":"2023-10-14T01:34:49.952083+0000","last_fullsized":"2023-10-14T01:34:49.952083+0000","mapping_epoch":15,"log_start":"137'603036","ondisk_log_start":"137'603036","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7483166720,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":608700,"num_read_kb":2402068,"num_write":607885,"num_write_kb":2431540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'4072","reported_seq":5930,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:49.067454+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:34:49.067454+0000","last_peered":"2023-10-14T01:34:49.067454+0000","last_clean":"2023-10-14T01:34:49.067454+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:34:49.067454+0000","last_undegraded":"2023-10-14T01:34:49.067454+0000","last_fullsized":"2023-10-14T01:34:49.067454+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'582006","reported_seq":1157845,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:34:50.011893+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:34:50.011893+0000","last_peered":"2023-10-14T01:34:50.011893+0000","last_clean":"2023-10-14T01:34:50.011893+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:34:50.011893+0000","last_undegraded":"2023-10-14T01:34:50.011893+0000","last_fullsized":"2023-10-14T01:34:50.011893+0000","mapping_epoch":15,"log_start":"137'571929","ondisk_log_start":"137'571929","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3577,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 790s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7106744320,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":577394,"num_read_kb":2278016,"num_write":577150,"num_write_kb":2308600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3372,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:46.255565+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:34:46.255565+0000","last_peered":"2023-10-14T01:34:46.255565+0000","last_clean":"2023-10-14T01:34:46.255565+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:34:46.255565+0000","last_undegraded":"2023-10-14T01:34:46.255565+0000","last_fullsized":"2023-10-14T01:34:46.255565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'604262","reported_seq":1202760,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:49.951370+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:34:49.951370+0000","last_peered":"2023-10-14T01:34:49.951370+0000","last_clean":"2023-10-14T01:34:49.951370+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:34:49.951370+0000","last_undegraded":"2023-10-14T01:34:49.951370+0000","last_fullsized":"2023-10-14T01:34:49.951370+0000","mapping_epoch":15,"log_start":"137'594228","ondisk_log_start":"137'594228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7364276224,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":600345,"num_read_kb":2368952,"num_write":599200,"num_write_kb":2396800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3523,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:46.259246+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:34:46.259246+0000","last_peered":"2023-10-14T01:34:46.259246+0000","last_clean":"2023-10-14T01:34:46.259246+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:34:46.259246+0000","last_undegraded":"2023-10-14T01:34:46.259246+0000","last_fullsized":"2023-10-14T01:34:46.259246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'574845","reported_seq":1143812,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:50.009176+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:34:50.009176+0000","last_peered":"2023-10-14T01:34:50.009176+0000","last_clean":"2023-10-14T01:34:50.009176+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:34:50.009176+0000","last_undegraded":"2023-10-14T01:34:50.009176+0000","last_fullsized":"2023-10-14T01:34:50.009176+0000","mapping_epoch":15,"log_start":"137'564827","ondisk_log_start":"137'564827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":7007490067,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":570731,"num_read_kb":2251296,"num_write":570021,"num_write_kb":2280078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"138'11661","reported_seq":16531,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:48.933887+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:34:48.933887+0000","last_peered":"2023-10-14T01:34:48.933887+0000","last_clean":"2023-10-14T01:34:48.933887+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:34:48.933887+0000","last_undegraded":"2023-10-14T01:34:48.933887+0000","last_fullsized":"2023-10-14T01:34:48.933887+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":320,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.612203+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:33:44.612203+0000","last_peered":"2023-10-14T01:33:44.612203+0000","last_clean":"2023-10-14T01:33:44.612203+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:33:44.612203+0000","last_undegraded":"2023-10-14T01:33:44.612203+0000","last_fullsized":"2023-10-14T01:33:44.612203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28961677331,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2357170,"num_read_kb":9300332,"num_write":2354256,"num_write_kb":9417018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19282399232,"data_stored":28464762918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40142,"ondisk_log_size":40142,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":50145,"ondisk_log_size":50145,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739446,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176304,"kb_used_data":9414632,"kb_used_omap":1482,"kb_used_meta":1760181,"kb_avail":83195536,"statfs":{"total":96636764160,"available":85192228864,"internally_reserved":0,"allocated":9640583168,"data_stored":14230882711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1802426339},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:34:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.42099999999999999,"15min":0.40100000000000002},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":1.5740000000000001,"5min":4.1369999999999996,"15min":8.8360000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42599999999999999,"15min":0.39000000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.193},"max":{"1min":2.1989999999999998,"5min":3.1749999999999998,"15min":7.9820000000000002},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:34:35 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.33800000000000002,"15min":0.44400000000000001},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47499999999999998,"5min":0.82199999999999995,"15min":11.952},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":3.028,"5min":3.028,"15min":12.029999999999999},"last":0.40899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739445,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6201452,"kb_used_data":4590048,"kb_used_omap":626,"kb_used_meta":1610765,"kb_avail":88170388,"statfs":{"total":96636764160,"available":90286477312,"internally_reserved":0,"allocated":4700209152,"data_stored":6940583451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":641763,"internal_metadata":1649423645},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,3,10,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:34:29 2023","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":0.63800000000000001,"15min":0.52000000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":24.853000000000002,"5min":24.853000000000002,"15min":24.853000000000002},"last":0.19500000000000001},{"interface":"front","average":{"1min":1.643,"5min":0.69499999999999995,"15min":0.56200000000000006},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.192},"max":{"1min":24.898,"5min":24.898,"15min":24.898},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:34:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.42499999999999999,"15min":0.41099999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.3739999999999997,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":6.2800000000000002,"15min":6.2800000000000002},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5933184,"kb_used_data":4829624,"kb_used_omap":747,"kb_used_meta":1102804,"kb_avail":88438656,"statfs":{"total":96636764160,"available":90561183744,"internally_reserved":0,"allocated":4945534976,"data_stored":7294311864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":1129271376},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.52700000000000002,"15min":0.48899999999999999},"min":{"1min":0.20499999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.65800000000000003,"5min":5.7549999999999999,"15min":16.023},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.22900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.624,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.251}]},{"osd":2,"last update":"Sat Oct 14 01:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39500000000000002,"15min":0.45500000000000002},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.17699999999999999},"max":{"1min":1.5629999999999999,"5min":2.8069999999999999,"15min":5.141},"last":0.371},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.43099999999999999,"15min":0.41899999999999998},"min":{"1min":0.22800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.80200000000000005,"5min":4.673,"15min":5.0609999999999999},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4944224256,"data_stored":7293952000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4698808320,"data_stored":6940262419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9639366656,"data_stored":14230548499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:34:51.795 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:34:51.795 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:35:11.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:35:12.287 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:35:12.288 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:35:12.314 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2837,"stamp":"2023-10-14T01:35:11.704647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28995391205,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2399100,"num_read_kb":9361782,"num_write":2363589,"num_write_kb":9453415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90535,"ondisk_log_size":90535,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23587140,"kb_used_data":18905900,"kb_used_omap":2856,"kb_used_meta":4678359,"kb_avail":259528380,"statfs":{"total":289910292480,"available":265757061120,"internally_reserved":0,"allocated":19359641600,"data_stored":28539092330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2925232,"internal_metadata":4790639952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,22,1,2,8,10,17,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24883200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6101,"num_read_kb":24336,"num_write":6415,"num_write_kb":25660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002171"},"pg_stats":[{"pgid":"2.7","version":"138'7888","reported_seq":11422,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:08.260735+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:35:08.260735+0000","last_peered":"2023-10-14T01:35:08.260735+0000","last_clean":"2023-10-14T01:35:08.260735+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:35:08.260735+0000","last_undegraded":"2023-10-14T01:35:08.260735+0000","last_fullsized":"2023-10-14T01:35:08.260735+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7888,"log_dups_size":0,"ondisk_log_size":7888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'11757","reported_seq":18226,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:08.268783+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:35:08.268783+0000","last_peered":"2023-10-14T01:35:08.268783+0000","last_clean":"2023-10-14T01:35:08.268783+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:35:08.268783+0000","last_undegraded":"2023-10-14T01:35:08.268783+0000","last_fullsized":"2023-10-14T01:35:08.268783+0000","mapping_epoch":12,"log_start":"58'1700","ondisk_log_start":"58'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'7874","reported_seq":11207,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:08.259312+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:35:08.259312+0000","last_peered":"2023-10-14T01:35:08.259312+0000","last_clean":"2023-10-14T01:35:08.259312+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:35:08.259312+0000","last_undegraded":"2023-10-14T01:35:08.259312+0000","last_fullsized":"2023-10-14T01:35:08.259312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'19401","reported_seq":28167,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:08.287364+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:35:08.287364+0000","last_peered":"2023-10-14T01:35:08.287364+0000","last_clean":"2023-10-14T01:35:08.287364+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:35:08.287364+0000","last_undegraded":"2023-10-14T01:35:08.287364+0000","last_fullsized":"2023-10-14T01:35:08.287364+0000","mapping_epoch":12,"log_start":"114'9400","ondisk_log_start":"114'9400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'615262","reported_seq":1224195,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:10.039129+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:35:10.039129+0000","last_peered":"2023-10-14T01:35:10.039129+0000","last_clean":"2023-10-14T01:35:10.039129+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:35:10.039129+0000","last_undegraded":"2023-10-14T01:35:10.039129+0000","last_fullsized":"2023-10-14T01:35:10.039129+0000","mapping_epoch":15,"log_start":"137'605236","ondisk_log_start":"137'605236","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7491768320,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":610826,"num_read_kb":2410544,"num_write":610098,"num_write_kb":2440392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"138'4085","reported_seq":5947,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:08.486326+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:35:08.486326+0000","last_peered":"2023-10-14T01:35:08.486326+0000","last_clean":"2023-10-14T01:35:08.486326+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:35:08.486326+0000","last_undegraded":"2023-10-14T01:35:08.486326+0000","last_fullsized":"2023-10-14T01:35:08.486326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'584112","reported_seq":1161985,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:35:10.079162+0000","last_change":"2023-10-14T01:21:39.650961+0000","last_active":"2023-10-14T01:35:10.079162+0000","last_peered":"2023-10-14T01:35:10.079162+0000","last_clean":"2023-10-14T01:35:10.079162+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:35:10.079162+0000","last_undegraded":"2023-10-14T01:35:10.079162+0000","last_fullsized":"2023-10-14T01:35:10.079162+0000","mapping_epoch":15,"log_start":"137'574030","ondisk_log_start":"137'574030","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:05:21.070436+0000","last_clean_scrub_stamp":"2023-10-14T00:05:21.070436+0000","objects_scrubbed":3628,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 810s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7114801152,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":579421,"num_read_kb":2286100,"num_write":579221,"num_write_kb":2316884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3381,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:06.735097+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:35:06.735097+0000","last_peered":"2023-10-14T01:35:06.735097+0000","last_clean":"2023-10-14T01:35:06.735097+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:35:06.735097+0000","last_undegraded":"2023-10-14T01:35:06.735097+0000","last_fullsized":"2023-10-14T01:35:06.735097+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'606463","reported_seq":1207010,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:10.037514+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:35:10.037514+0000","last_peered":"2023-10-14T01:35:10.037514+0000","last_clean":"2023-10-14T01:35:10.037514+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:35:10.037514+0000","last_undegraded":"2023-10-14T01:35:10.037514+0000","last_fullsized":"2023-10-14T01:35:10.037514+0000","mapping_epoch":15,"log_start":"137'596428","ondisk_log_start":"137'596428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7372734464,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":602395,"num_read_kb":2377132,"num_write":601401,"num_write_kb":2405604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3531,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:06.740556+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:35:06.740556+0000","last_peered":"2023-10-14T01:35:06.740556+0000","last_clean":"2023-10-14T01:35:06.740556+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:35:06.740556+0000","last_undegraded":"2023-10-14T01:35:06.740556+0000","last_fullsized":"2023-10-14T01:35:06.740556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'576897","reported_seq":1147840,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:10.083239+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:35:10.083239+0000","last_peered":"2023-10-14T01:35:10.083239+0000","last_clean":"2023-10-14T01:35:10.083239+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:35:10.083239+0000","last_undegraded":"2023-10-14T01:35:10.083239+0000","last_fullsized":"2023-10-14T01:35:10.083239+0000","mapping_epoch":15,"log_start":"137'566827","ondisk_log_start":"137'566827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":7015493651,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":572710,"num_read_kb":2259200,"num_write":572073,"num_write_kb":2288286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"138'11694","reported_seq":16576,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:35:08.312230+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:35:08.312230+0000","last_peered":"2023-10-14T01:35:08.312230+0000","last_clean":"2023-10-14T01:35:08.312230+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:35:08.312230+0000","last_undegraded":"2023-10-14T01:35:08.312230+0000","last_fullsized":"2023-10-14T01:35:08.312230+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":320,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:33:44.612203+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:33:44.612203+0000","last_peered":"2023-10-14T01:33:44.612203+0000","last_clean":"2023-10-14T01:33:44.612203+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:33:44.612203+0000","last_undegraded":"2023-10-14T01:33:44.612203+0000","last_fullsized":"2023-10-14T01:33:44.612203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28994797587,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2365352,"num_read_kb":9332976,"num_write":2362793,"num_write_kb":9451166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19355824128,"data_stored":28538187814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":50223,"ondisk_log_size":50223,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739451,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11321944,"kb_used_data":9452048,"kb_used_omap":1482,"kb_used_meta":1868405,"kb_avail":83049896,"statfs":{"total":96636764160,"available":85043093504,"internally_reserved":0,"allocated":9678897152,"data_stored":14269196695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":1913247715},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:35:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.47499999999999998,"15min":0.42699999999999999},"min":{"1min":0.185,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":8.8360000000000003},"last":0.309},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.40799999999999997,"15min":0.40699999999999997},"min":{"1min":0.19900000000000001,"5min":0.19600000000000001,"15min":0.193},"max":{"1min":5.1580000000000004,"5min":5.1580000000000004,"15min":7.9820000000000002},"last":0.26100000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:34:35 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.33800000000000002,"15min":0.44400000000000001},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47499999999999998,"5min":0.82199999999999995,"15min":11.952},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":3.028,"5min":3.028,"15min":12.029999999999999},"last":0.501}]}]},{"osd":1,"up_from":8,"seq":34359739449,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6242212,"kb_used_data":4606620,"kb_used_omap":626,"kb_used_meta":1634957,"kb_avail":88129628,"statfs":{"total":96636764160,"available":90244739072,"internally_reserved":0,"allocated":4717178880,"data_stored":6957553179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":641763,"internal_metadata":1674196253},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,3,8,10,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:34:29 2023","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":0.63800000000000001,"15min":0.52000000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":24.853000000000002,"5min":24.853000000000002,"15min":24.853000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.643,"5min":0.69499999999999995,"15min":0.56200000000000006},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.192},"max":{"1min":24.898,"5min":24.898,"15min":24.898},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Oct 14 01:34:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.42499999999999999,"15min":0.41099999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.3739999999999997,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":6.2800000000000002,"15min":6.2800000000000002},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739449,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6022984,"kb_used_data":4847232,"kb_used_omap":747,"kb_used_meta":1174996,"kb_avail":88348856,"statfs":{"total":96636764160,"available":90469228544,"internally_reserved":0,"allocated":4963565568,"data_stored":7312342456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":1203195984},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,22,0,0,1,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.52700000000000002,"15min":0.48899999999999999},"min":{"1min":0.20499999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.65800000000000003,"5min":5.7549999999999999,"15min":16.023},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.22900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.624,"5min":5.1909999999999998,"15min":16.065000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39500000000000002,"15min":0.45500000000000002},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.17699999999999999},"max":{"1min":1.5629999999999999,"5min":2.8069999999999999,"15min":5.141},"last":0.318},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.43099999999999999,"15min":0.41899999999999998},"min":{"1min":0.22800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.80200000000000005,"5min":4.673,"15min":5.0609999999999999},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4962316288,"data_stored":7312044032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4715708416,"data_stored":6957162515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9677799424,"data_stored":14268981267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:35:12.316 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:05:21.070436+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=5, tm_sec=21, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=11, tm_sec=36, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:35:12.317 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:35:30.832 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95cf2bc70 2023-10-14T01:35:30.832 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d1c0b60 2023-10-14T01:35:30.833 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc340 2023-10-14T01:35:30.833 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cc820 2023-10-14T01:35:30.833 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cd110 2023-10-14T01:35:30.834 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdad0 2023-10-14T01:35:30.834 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d8cdba0 2023-10-14T01:35:30.834 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943790 2023-10-14T01:35:30.834 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95d943930 2023-10-14T01:35:30.835 INFO:tasks.ceph.osd.1.smithi081.stderr:2023-10-14T01:35:30.830+0000 7ff6372e3640 -1 reset not still connected to 0x55b95e1f0d00 2023-10-14T01:35:32.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:35:32.903 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-10-14T01:35:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2023-10-14T01:35:32.941 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":2848,"stamp":"2023-10-14T01:35:31.708073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29029687013,"num_objects":14952,"num_object_clones":9728,"num_object_copies":29904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14952,"num_whiteouts":0,"num_read":2407651,"num_read_kb":9395930,"num_write":2372422,"num_write_kb":9488747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90633,"ondisk_log_size":90633,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23875776,"kb_used_data":18976232,"kb_used_omap":2856,"kb_used_meta":4896663,"kb_avail":259239744,"statfs":{"total":289910292480,"available":265461497856,"internally_reserved":0,"allocated":19431661568,"data_stored":28611099353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2925232,"internal_metadata":5014183248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,4,3,7,14,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17342464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4304,"num_read_kb":17192,"num_write":4456,"num_write_kb":17824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001755"},"pg_stats":[{"pgid":"2.7","version":"139'7906","reported_seq":11449,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:29.722617+0000","last_change":"2023-10-14T01:11:58.599814+0000","last_active":"2023-10-14T01:35:29.722617+0000","last_peered":"2023-10-14T01:35:29.722617+0000","last_clean":"2023-10-14T01:35:29.722617+0000","last_became_active":"2023-10-14T00:05:17.001930+0000","last_became_peered":"2023-10-14T00:05:17.001930+0000","last_unstale":"2023-10-14T01:35:29.722617+0000","last_undegraded":"2023-10-14T01:35:29.722617+0000","last_fullsized":"2023-10-14T01:35:29.722617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'6063","last_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_deep_scrub":"119'6063","last_deep_scrub_stamp":"2023-10-14T01:11:58.599609+0000","last_clean_scrub_stamp":"2023-10-14T01:11:58.599609+0000","objects_scrubbed":13,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:21:47.844673+0000","scrub_duration":0.064248681000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3127,"num_read_kb":2616,"num_write":89,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'11784","reported_seq":18266,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:29.728169+0000","last_change":"2023-10-14T01:11:47.653765+0000","last_active":"2023-10-14T01:35:29.728169+0000","last_peered":"2023-10-14T01:35:29.728169+0000","last_clean":"2023-10-14T01:35:29.728169+0000","last_became_active":"2023-10-14T00:05:17.002471+0000","last_became_peered":"2023-10-14T00:05:17.002471+0000","last_unstale":"2023-10-14T01:35:29.728169+0000","last_undegraded":"2023-10-14T01:35:29.728169+0000","last_fullsized":"2023-10-14T01:35:29.728169+0000","mapping_epoch":12,"log_start":"58'1700","ondisk_log_start":"58'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8992","last_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_deep_scrub":"118'8992","last_deep_scrub_stamp":"2023-10-14T01:11:47.653581+0000","last_clean_scrub_stamp":"2023-10-14T01:11:47.653581+0000","objects_scrubbed":10,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:15:23.974077+0000","scrub_duration":0.10005247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7413,"num_read_kb":6656,"num_write":118,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3616,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'7892","reported_seq":11235,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:29.714711+0000","last_change":"2023-10-14T01:11:44.722869+0000","last_active":"2023-10-14T01:35:29.714711+0000","last_peered":"2023-10-14T01:35:29.714711+0000","last_clean":"2023-10-14T01:35:29.714711+0000","last_became_active":"2023-10-14T00:05:17.001882+0000","last_became_peered":"2023-10-14T00:05:17.001882+0000","last_unstale":"2023-10-14T01:35:29.714711+0000","last_undegraded":"2023-10-14T01:35:29.714711+0000","last_fullsized":"2023-10-14T01:35:29.714711+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6010","last_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_deep_scrub":"118'6010","last_deep_scrub_stamp":"2023-10-14T01:11:44.722683+0000","last_clean_scrub_stamp":"2023-10-14T01:11:44.722683+0000","objects_scrubbed":14,"log_size":7892,"log_dups_size":0,"ondisk_log_size":7892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:24:08.632794+0000","scrub_duration":0.084652330999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3032,"num_read_kb":2523,"num_write":58,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'19446","reported_seq":28234,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:29.742765+0000","last_change":"2023-10-14T01:11:59.699072+0000","last_active":"2023-10-14T01:35:29.742765+0000","last_peered":"2023-10-14T01:35:29.742765+0000","last_clean":"2023-10-14T01:35:29.742765+0000","last_became_active":"2023-10-14T00:05:17.005434+0000","last_became_peered":"2023-10-14T00:05:17.005434+0000","last_unstale":"2023-10-14T01:35:29.742765+0000","last_undegraded":"2023-10-14T01:35:29.742765+0000","last_fullsized":"2023-10-14T01:35:29.742765+0000","mapping_epoch":12,"log_start":"114'9400","ondisk_log_start":"114'9400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'14871","last_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_deep_scrub":"119'14871","last_deep_scrub_stamp":"2023-10-14T01:11:59.698057+0000","last_clean_scrub_stamp":"2023-10-14T01:11:59.698057+0000","objects_scrubbed":15,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:41:02.400005+0000","scrub_duration":0.15405618500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":8284,"num_read_kb":7018,"num_write":140,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4204,"num_omap_keys":100,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'617636","reported_seq":1228717,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:30.184341+0000","last_change":"2023-10-14T01:23:49.821500+0000","last_active":"2023-10-14T01:35:30.184341+0000","last_peered":"2023-10-14T01:35:30.184341+0000","last_clean":"2023-10-14T01:35:30.184341+0000","last_became_active":"2023-10-14T00:05:22.608640+0000","last_became_peered":"2023-10-14T00:05:22.608640+0000","last_unstale":"2023-10-14T01:35:30.184341+0000","last_undegraded":"2023-10-14T01:35:30.184341+0000","last_fullsized":"2023-10-14T01:35:30.184341+0000","mapping_epoch":15,"log_start":"137'607536","ondisk_log_start":"137'607536","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'542092","last_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_deep_scrub":"129'542092","last_deep_scrub_stamp":"2023-10-14T01:23:49.821043+0000","last_clean_scrub_stamp":"2023-10-14T01:23:49.821043+0000","objects_scrubbed":3842,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":725,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:23:17.958174+0000","scrub_duration":724.69211624100001,"objects_trimmed":0,"snaptrim_duration":0.01851947,"stat_sum":{"num_bytes":7501070336,"num_objects":3842,"num_object_clones":2520,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":612980,"num_read_kb":2419144,"num_write":612472,"num_write_kb":2449888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"139'4099","reported_seq":5967,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:27.219203+0000","last_change":"2023-10-14T01:11:41.299313+0000","last_active":"2023-10-14T01:35:27.219203+0000","last_peered":"2023-10-14T01:35:27.219203+0000","last_clean":"2023-10-14T01:35:27.219203+0000","last_became_active":"2023-10-14T00:05:16.999492+0000","last_became_peered":"2023-10-14T00:05:16.999492+0000","last_unstale":"2023-10-14T01:35:27.219203+0000","last_undegraded":"2023-10-14T01:35:27.219203+0000","last_fullsized":"2023-10-14T01:35:27.219203+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'3132","last_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_deep_scrub":"118'3132","last_deep_scrub_stamp":"2023-10-14T01:11:41.299055+0000","last_clean_scrub_stamp":"2023-10-14T01:11:41.299055+0000","objects_scrubbed":11,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:48:31.138523+0000","scrub_duration":0.18690166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1601,"num_read_kb":1366,"num_write":56,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2095,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'586271","reported_seq":1166307,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:30.160577+0000","last_change":"2023-10-14T01:35:13.639758+0000","last_active":"2023-10-14T01:35:30.160577+0000","last_peered":"2023-10-14T01:35:30.160577+0000","last_clean":"2023-10-14T01:35:30.160577+0000","last_became_active":"2023-10-14T00:05:22.608630+0000","last_became_peered":"2023-10-14T00:05:22.608630+0000","last_unstale":"2023-10-14T01:35:30.160577+0000","last_undegraded":"2023-10-14T01:35:30.160577+0000","last_fullsized":"2023-10-14T01:35:30.160577+0000","mapping_epoch":15,"log_start":"137'576230","ondisk_log_start":"137'576230","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'584468","last_scrub_stamp":"2023-10-14T01:35:13.639556+0000","last_deep_scrub":"138'584468","last_deep_scrub_stamp":"2023-10-14T01:35:13.639556+0000","last_clean_scrub_stamp":"2023-10-14T01:35:13.639556+0000","objects_scrubbed":3651,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":814,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:25:15.675712+0000","scrub_duration":813.98742935099995,"objects_trimmed":0,"snaptrim_duration":0.0010030169999999999,"stat_sum":{"num_bytes":7123152896,"num_objects":3651,"num_object_clones":2396,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":581577,"num_read_kb":2294708,"num_write":581364,"num_write_kb":2325456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"118'94","reported_seq":3390,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:27.219214+0000","last_change":"2023-10-14T01:11:40.146832+0000","last_active":"2023-10-14T01:35:27.219214+0000","last_peered":"2023-10-14T01:35:27.219214+0000","last_clean":"2023-10-14T01:35:27.219214+0000","last_became_active":"2023-10-14T00:05:16.997851+0000","last_became_peered":"2023-10-14T00:05:16.997851+0000","last_unstale":"2023-10-14T01:35:27.219214+0000","last_undegraded":"2023-10-14T01:35:27.219214+0000","last_fullsized":"2023-10-14T01:35:27.219214+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'94","last_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_deep_scrub":"118'94","last_deep_scrub_stamp":"2023-10-14T01:11:40.146649+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.146649+0000","objects_scrubbed":10,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:26:07.320074+0000","scrub_duration":0.048899982000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2952,"num_read_kb":2469,"num_write":50,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'608692","reported_seq":1211410,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:30.166273+0000","last_change":"2023-10-14T01:22:42.425428+0000","last_active":"2023-10-14T01:35:30.166273+0000","last_peered":"2023-10-14T01:35:30.166273+0000","last_clean":"2023-10-14T01:35:30.166273+0000","last_became_active":"2023-10-14T00:05:22.608849+0000","last_became_peered":"2023-10-14T00:05:22.608849+0000","last_unstale":"2023-10-14T01:35:30.166273+0000","last_undegraded":"2023-10-14T01:35:30.166273+0000","last_fullsized":"2023-10-14T01:35:30.166273+0000","mapping_epoch":15,"log_start":"137'598628","ondisk_log_start":"137'598628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'528049","last_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_deep_scrub":"128'528049","last_deep_scrub_stamp":"2023-10-14T01:22:42.425067+0000","last_clean_scrub_stamp":"2023-10-14T01:22:42.425067+0000","objects_scrubbed":3765,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":659,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:42:00.727122+0000","scrub_duration":658.29182614399997,"objects_trimmed":0,"snaptrim_duration":0.083982343000000001,"stat_sum":{"num_bytes":7381319680,"num_objects":3765,"num_object_clones":2460,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":604566,"num_read_kb":2385800,"num_write":603630,"num_write_kb":2414520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"118'130","reported_seq":3540,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:27.219093+0000","last_change":"2023-10-14T01:11:40.961628+0000","last_active":"2023-10-14T01:35:27.219093+0000","last_peered":"2023-10-14T01:35:27.219093+0000","last_clean":"2023-10-14T01:35:27.219093+0000","last_became_active":"2023-10-14T00:05:17.000770+0000","last_became_peered":"2023-10-14T00:05:17.000770+0000","last_unstale":"2023-10-14T01:35:27.219093+0000","last_undegraded":"2023-10-14T01:35:27.219093+0000","last_fullsized":"2023-10-14T01:35:27.219093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'130","last_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_deep_scrub":"118'130","last_deep_scrub_stamp":"2023-10-14T01:11:40.961435+0000","last_clean_scrub_stamp":"2023-10-14T01:11:40.961435+0000","objects_scrubbed":11,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:56:08.973480+0000","scrub_duration":0.036265288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2763,"num_read_kb":2330,"num_write":63,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1036,"num_omap_keys":22,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'578984","reported_seq":1152001,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:30.157500+0000","last_change":"2023-10-14T01:21:37.822078+0000","last_active":"2023-10-14T01:35:30.157500+0000","last_peered":"2023-10-14T01:35:30.157500+0000","last_clean":"2023-10-14T01:35:30.157500+0000","last_became_active":"2023-10-14T00:05:22.608655+0000","last_became_peered":"2023-10-14T00:05:22.608655+0000","last_unstale":"2023-10-14T01:35:30.157500+0000","last_undegraded":"2023-10-14T01:35:30.157500+0000","last_fullsized":"2023-10-14T01:35:30.157500+0000","mapping_epoch":15,"log_start":"137'568927","ondisk_log_start":"137'568927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'495997","last_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_deep_scrub":"127'495997","last_deep_scrub_stamp":"2023-10-14T01:21:37.821903+0000","last_clean_scrub_stamp":"2023-10-14T01:21:37.821903+0000","objects_scrubbed":3591,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":573,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:21.150701+0000","scrub_duration":572.27552990900006,"objects_trimmed":0,"snaptrim_duration":0.0086752129999999993,"stat_sum":{"num_bytes":7023550483,"num_objects":3591,"num_object_clones":2352,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":574780,"num_read_kb":2267472,"num_write":574160,"num_write_kb":2296634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"139'11721","reported_seq":16617,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:35:29.759835+0000","last_change":"2023-10-14T01:11:46.798926+0000","last_active":"2023-10-14T01:35:29.759835+0000","last_peered":"2023-10-14T01:35:29.759835+0000","last_clean":"2023-10-14T01:35:29.759835+0000","last_became_active":"2023-10-14T00:05:17.003822+0000","last_became_peered":"2023-10-14T00:05:17.003822+0000","last_unstale":"2023-10-14T01:35:29.759835+0000","last_undegraded":"2023-10-14T01:35:29.759835+0000","last_fullsized":"2023-10-14T01:35:29.759835+0000","mapping_epoch":12,"log_start":"59'1700","ondisk_log_start":"59'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8937","last_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_deep_scrub":"118'8937","last_deep_scrub_stamp":"2023-10-14T01:11:46.798764+0000","last_clean_scrub_stamp":"2023-10-14T01:11:46.798764+0000","objects_scrubbed":17,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:25:44.543922+0000","scrub_duration":0.238290101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4470,"num_read_kb":3737,"num_write":85,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"118'99","reported_seq":321,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:34:16.583417+0000","last_change":"2023-10-14T01:11:43.787781+0000","last_active":"2023-10-14T01:34:16.583417+0000","last_peered":"2023-10-14T01:34:16.583417+0000","last_clean":"2023-10-14T01:34:16.583417+0000","last_became_active":"2023-10-14T00:05:12.382330+0000","last_became_peered":"2023-10-14T00:05:12.382330+0000","last_unstale":"2023-10-14T01:34:16.583417+0000","last_undegraded":"2023-10-14T01:34:16.583417+0000","last_fullsized":"2023-10-14T01:34:16.583417+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'99","last_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_deep_scrub":"118'99","last_deep_scrub_stamp":"2023-10-14T01:11:43.780955+0000","last_clean_scrub_stamp":"2023-10-14T01:11:43.780955+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:04:22.304710+0000","scrub_duration":0.104887749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29029093395,"num_objects":14849,"num_object_clones":9728,"num_object_copies":29698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14849,"num_whiteouts":0,"num_read":2373903,"num_read_kb":9367124,"num_write":2371626,"num_write_kb":9486498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19428089856,"data_stored":28610453542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40262,"ondisk_log_size":40262,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3250,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":33642,"num_read_kb":28715,"num_write":659,"num_write_kb":355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17566,"num_omap_keys":341,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":6500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38876,"internal_metadata":0},"log_size":50272,"ondisk_log_size":50272,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739455,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11456528,"kb_used_data":9487048,"kb_used_omap":1482,"kb_used_meta":1967989,"kb_avail":82915312,"statfs":{"total":96636764160,"available":84905279488,"internally_reserved":0,"allocated":9714737152,"data_stored":14305032380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1517597,"internal_metadata":2015221731},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:35:09 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.47499999999999998,"15min":0.42699999999999999},"min":{"1min":0.185,"5min":0.17799999999999999,"15min":0.16900000000000001},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":8.8360000000000003},"last":0.214},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.40799999999999997,"15min":0.40699999999999997},"min":{"1min":0.19900000000000001,"5min":0.19600000000000001,"15min":0.193},"max":{"1min":5.1580000000000004,"5min":5.1580000000000004,"15min":7.9820000000000002},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:34:35 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.33800000000000002,"15min":0.44400000000000001},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47499999999999998,"5min":0.82199999999999995,"15min":11.952},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.51500000000000001,"15min":0.55900000000000005},"min":{"1min":0.23499999999999999,"5min":0.187,"15min":0.18099999999999999},"max":{"1min":3.028,"5min":3.028,"15min":12.029999999999999},"last":0.29799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739453,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6313328,"kb_used_data":4623592,"kb_used_omap":626,"kb_used_meta":1689101,"kb_avail":88058512,"statfs":{"total":96636764160,"available":90171916288,"internally_reserved":0,"allocated":4734558208,"data_stored":6974928192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":641763,"internal_metadata":1729639709},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:35:30 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.60899999999999999,"15min":0.52200000000000002},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.16900000000000001},"max":{"1min":2.3069999999999999,"5min":24.853000000000002,"15min":24.853000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.66300000000000003,"15min":0.56299999999999994},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.192},"max":{"1min":2.3439999999999999,"5min":24.898,"15min":24.898},"last":0.25600000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:35:30 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.35099999999999998,"15min":0.40400000000000003},"min":{"1min":0.20499999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":5.3739999999999997,"15min":6.5199999999999996},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.434},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":0.877,"5min":0.877,"15min":6.2800000000000002},"last":0.379}]}]},{"osd":0,"up_from":8,"seq":34359739453,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6105920,"kb_used_data":4865592,"kb_used_omap":747,"kb_used_meta":1239572,"kb_avail":88265920,"statfs":{"total":96636764160,"available":90384302080,"internally_reserved":0,"allocated":4982366208,"data_stored":7331138781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":765872,"internal_metadata":1269321808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,4,12,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.48099999999999998},"min":{"1min":0.17299999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.58999999999999997,"5min":5.7549999999999999,"15min":16.023},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.47999999999999998},"min":{"1min":0.23400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8340000000000001,"5min":4.6360000000000001,"15min":16.065000000000001},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38700000000000001,"15min":0.45400000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.17699999999999999},"max":{"1min":2.6110000000000002,"5min":2.8069999999999999,"15min":5.141},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.45400000000000001,"15min":0.41599999999999998},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":2.8780000000000001,"5min":4.673,"15min":5.0609999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13670,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":3250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17434,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":1359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7772,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4981166080,"data_stored":7330893824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4733022208,"data_stored":6974476307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9713901568,"data_stored":14305083411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:35:32.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-14T01:35:33.459 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-14T01:35:33.460 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-14T01:35:33.460 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-14T01:35:33.461 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:35:38.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-14T01:35:39.563 INFO:tasks.ceph.osd.0:Stopped 2023-10-14T01:35:39.564 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-14T01:35:39.564 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:35:43.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-14T01:35:43.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-14T01:35:45.666 INFO:tasks.ceph.osd.1:Stopped 2023-10-14T01:35:45.667 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-14T01:35:45.667 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:35:48.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-14T01:35:48.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-14T01:35:51.769 INFO:tasks.ceph.osd.2:Stopped 2023-10-14T01:35:51.770 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-14T01:35:51.770 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-14T01:35:51.771 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:35:53.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-14T01:35:53.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-14T01:35:57.872 INFO:tasks.ceph.mgr.x:Stopped 2023-10-14T01:35:57.873 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-14T01:35:57.874 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-14T01:35:57.874 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:35:58.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-14T01:35:58.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-14T01:36:03.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~5s 2023-10-14T01:36:03.976 INFO:tasks.ceph.mon.a:Stopped 2023-10-14T01:36:03.977 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-14T01:36:03.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-14T01:36:04.046 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T01:36:04.047 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-14T01:36:05.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T01:36:05.390 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-14T01:36:05.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T01:36:05.686 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-14T01:36:06.015 INFO:tasks.ceph:Archiving mon data... 2023-10-14T01:36:06.017 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279/data/mon.a.tgz 2023-10-14T01:36:06.018 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2023-10-14T01:36:06.047 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.zlIH2MdNuT 2023-10-14T01:36:06.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zlIH2MdNuT 2023-10-14T01:36:06.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.zlIH2MdNuT 2023-10-14T01:36:06.311 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.zlIH2MdNuT is 606KB 2023-10-14T01:36:06.371 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.zlIH2MdNuT 2023-10-14T01:36:06.376 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-14T01:36:06.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T01:36:06.471 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-14T01:36:06.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T01:36:06.498 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T01:36:06.509 INFO:tasks.ceph:Archiving crash dumps... 2023-10-14T01:36:06.511 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279/remote/smithi081/crash 2023-10-14T01:36:06.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T01:36:06.565 INFO:tasks.ceph:Compressing logs... 2023-10-14T01:36:06.566 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T01:36:06.637 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39489.log 2023-10-14T01:36:06.812 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50722.log 2023-10-14T01:36:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2023-10-14T01:36:06.813 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39489.log.gz 2023-10-14T01:36:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53004.log 2023-10-14T01:36:06.813 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50722.log.gz 2023-10-14T01:36:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2023-10-14T01:36:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.53004.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56740.log 2023-10-14T01:36:06.814 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53004.log.gz 2023-10-14T01:36:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46178.log 2023-10-14T01:36:06.815 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2023-10-14T01:36:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45686.log 2023-10-14T01:36:06.815 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56740.log.gz 2023-10-14T01:36:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55091.log 2023-10-14T01:36:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56528.log 2023-10-14T01:36:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55091.log.gz 2023-10-14T01:36:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49808.log 2023-10-14T01:36:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48333.log 2023-10-14T01:36:06.817 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56528.log.gz 2023-10-14T01:36:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2023-10-14T01:36:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40466.log 2023-10-14T01:36:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44081.log 2023-10-14T01:36:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40810.log: /var/log/ceph/ceph-client.admin.40466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53855.log 2023-10-14T01:36:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2023-10-14T01:36:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55254.log 2023-10-14T01:36:06.819 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53855.log.gz 2023-10-14T01:36:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2023-10-14T01:36:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49509.log: /var/log/ceph/ceph-client.admin.55254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40756.log 2023-10-14T01:36:06.820 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55254.log.gz 2023-10-14T01:36:06.820 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46527.log 2023-10-14T01:36:06.820 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48282.log 2023-10-14T01:36:06.821 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40756.log: /var/log/ceph/ceph-client.admin.46527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54219.log 2023-10-14T01:36:06.821 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51603.log 2023-10-14T01:36:06.821 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54474.log 2023-10-14T01:36:06.821 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54219.log.gz 2023-10-14T01:36:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44705.log 2023-10-14T01:36:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51603.log.gz 2023-10-14T01:36:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54124.log 2023-10-14T01:36:06.823 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54474.log.gz 2023-10-14T01:36:06.823 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51785.log 2023-10-14T01:36:06.823 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2023-10-14T01:36:06.823 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54124.log.gz 2023-10-14T01:36:06.824 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45164.log 2023-10-14T01:36:06.824 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51785.log.gz 2023-10-14T01:36:06.824 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40557.log 2023-10-14T01:36:06.824 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz 2023-10-14T01:36:06.825 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50571.log 2023-10-14T01:36:06.825 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48007.log 2023-10-14T01:36:06.825 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2023-10-14T01:36:06.825 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40466.log.gz 2023-10-14T01:36:06.826 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40756.log.gz 2023-10-14T01:36:06.826 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2023-10-14T01:36:06.826 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2023-10-14T01:36:06.827 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2023-10-14T01:36:06.827 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2023-10-14T01:36:06.827 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54292.log 2023-10-14T01:36:06.827 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46527.log.gz 2023-10-14T01:36:06.828 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44705.log.gz 2023-10-14T01:36:06.828 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48007.log.gz 2023-10-14T01:36:06.828 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46178.log.gz 0.0% 2023-10-14T01:36:06.828 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44081.log.gz 2023-10-14T01:36:06.829 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2023-10-14T01:36:06.829 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49808.log.gz 2023-10-14T01:36:06.829 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45686.log.gz 2023-10-14T01:36:06.829 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54311.log 0.0% 0.0% 2023-10-14T01:36:06.830 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48333.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2023-10-14T01:36:06.830 INFO:teuthology.orchestra.run.smithi081.stderr: 2023-10-14T01:36:06.830 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45164.log.gz 2023-10-14T01:36:06.831 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50571.log.gz 2023-10-14T01:36:06.831 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48282.log.gz 2023-10-14T01:36:06.831 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54510.log 2023-10-14T01:36:06.831 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54292.log.gz 2023-10-14T01:36:06.832 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53034.log 2023-10-14T01:36:06.832 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54311.log.gz 2023-10-14T01:36:06.832 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 2023-10-14T01:36:06.833 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54510.log.gz 2023-10-14T01:36:06.833 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41846.log 2023-10-14T01:36:06.833 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53034.log.gz 2023-10-14T01:36:06.833 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54981.log 2023-10-14T01:36:06.834 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2023-10-14T01:36:06.834 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50274.log 2023-10-14T01:36:06.834 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41846.log.gz 2023-10-14T01:36:06.834 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54383.log 2023-10-14T01:36:06.835 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54981.log.gz 2023-10-14T01:36:06.835 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52398.log 2023-10-14T01:36:06.835 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50274.log.gz 2023-10-14T01:36:06.836 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2023-10-14T01:36:06.836 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54383.log.gz 2023-10-14T01:36:06.836 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52488.log 2023-10-14T01:36:06.836 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52398.log.gz 2023-10-14T01:36:06.837 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2023-10-14T01:36:06.837 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2023-10-14T01:36:06.837 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54665.log 2023-10-14T01:36:06.837 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52488.log.gz 2023-10-14T01:36:06.838 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54201.log 2023-10-14T01:36:06.838 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2023-10-14T01:36:06.838 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49314.log 2023-10-14T01:36:06.839 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54665.log.gz 2023-10-14T01:36:06.839 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44575.log 2023-10-14T01:36:06.839 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54201.log.gz 2023-10-14T01:36:06.839 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41313.log 2023-10-14T01:36:06.840 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49314.log.gz 2023-10-14T01:36:06.840 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55197.log 2023-10-14T01:36:06.840 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44575.log.gz 2023-10-14T01:36:06.840 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2023-10-14T01:36:06.841 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41313.log.gz 2023-10-14T01:36:06.841 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54564.log 2023-10-14T01:36:06.841 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55197.log.gz 2023-10-14T01:36:06.842 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51693.log 2023-10-14T01:36:06.842 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2023-10-14T01:36:06.842 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2023-10-14T01:36:06.842 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54564.log.gz 2023-10-14T01:36:06.843 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55808.log 2023-10-14T01:36:06.843 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51693.log.gz 2023-10-14T01:36:06.843 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51483.log 2023-10-14T01:36:06.844 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2023-10-14T01:36:06.844 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57193.log 2023-10-14T01:36:06.844 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55808.log.gz 2023-10-14T01:36:06.844 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39429.log 2023-10-14T01:36:06.845 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51483.log.gz 2023-10-14T01:36:06.845 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57193.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-14T01:36:06.845 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57193.log.gz 2023-10-14T01:36:06.846 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45103.log 2023-10-14T01:36:06.846 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39429.log.gz 2023-10-14T01:36:06.846 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45380.log 2023-10-14T01:36:06.846 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.45103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53885.log 2023-10-14T01:36:06.847 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45103.log.gz 2023-10-14T01:36:06.847 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55782.log 2023-10-14T01:36:06.847 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45380.log.gz 2023-10-14T01:36:06.847 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49590.log 2023-10-14T01:36:06.848 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53885.log.gz 2023-10-14T01:36:06.848 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55782.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55782.log.gz -5 2023-10-14T01:36:06.848 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50172.log 2023-10-14T01:36:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43038.log 2023-10-14T01:36:06.849 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49590.log.gz 2023-10-14T01:36:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2023-10-14T01:36:06.849 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50172.log.gz 2023-10-14T01:36:06.850 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56606.log 2023-10-14T01:36:06.850 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43038.log.gz 2023-10-14T01:36:06.850 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48770.log 2023-10-14T01:36:06.851 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2023-10-14T01:36:06.851 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52638.log 2023-10-14T01:36:06.851 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56606.log.gz 2023-10-14T01:36:06.851 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2023-10-14T01:36:06.852 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48770.log.gz 2023-10-14T01:36:06.852 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45292.log 2023-10-14T01:36:06.852 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52638.log.gz 2023-10-14T01:36:06.852 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52248.log 2023-10-14T01:36:06.853 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2023-10-14T01:36:06.853 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52157.log 2023-10-14T01:36:06.853 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45292.log.gz 2023-10-14T01:36:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48990.log 2023-10-14T01:36:06.854 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52248.log.gz 2023-10-14T01:36:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53581.log 2023-10-14T01:36:06.854 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52157.log.gz 2023-10-14T01:36:06.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55569.log 2023-10-14T01:36:06.855 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48990.log.gz 2023-10-14T01:36:06.855 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41026.log 2023-10-14T01:36:06.855 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53581.log.gz 2023-10-14T01:36:06.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55332.log 2023-10-14T01:36:06.856 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55569.log.gz 2023-10-14T01:36:06.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46913.log 2023-10-14T01:36:06.857 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2023-10-14T01:36:06.857 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49784.log 2023-10-14T01:36:06.857 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55332.log.gz 2023-10-14T01:36:06.857 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53431.log 2023-10-14T01:36:06.858 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46913.log.gz 2023-10-14T01:36:06.858 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56475.log 2023-10-14T01:36:06.858 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49784.log.gz 2023-10-14T01:36:06.858 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46823.log 2023-10-14T01:36:06.859 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53431.log.gz 2023-10-14T01:36:06.859 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44517.log 2023-10-14T01:36:06.859 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56475.log.gz 2023-10-14T01:36:06.860 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2023-10-14T01:36:06.860 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46823.log.gz 2023-10-14T01:36:06.860 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54238.log 2023-10-14T01:36:06.861 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 2023-10-14T01:36:06.861 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2023-10-14T01:36:06.861 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2023-10-14T01:36:06.861 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53641.log 2023-10-14T01:36:06.862 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54238.log.gz 2023-10-14T01:36:06.862 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54717.log 2023-10-14T01:36:06.862 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2023-10-14T01:36:06.862 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2023-10-14T01:36:06.863 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53641.log.gz 2023-10-14T01:36:06.863 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43281.log 2023-10-14T01:36:06.863 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54717.log.gz 2023-10-14T01:36:06.864 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54795.log 2023-10-14T01:36:06.864 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2023-10-14T01:36:06.864 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43772.log 2023-10-14T01:36:06.864 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43281.log.gz 2023-10-14T01:36:06.865 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2023-10-14T01:36:06.865 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54795.log.gz 2023-10-14T01:36:06.865 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50936.log 2023-10-14T01:36:06.865 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43772.log.gz 2023-10-14T01:36:06.866 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2023-10-14T01:36:06.866 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2023-10-14T01:36:06.866 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49755.log 2023-10-14T01:36:06.867 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50936.log.gz 2023-10-14T01:36:06.867 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42485.log 2023-10-14T01:36:06.867 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2023-10-14T01:36:06.867 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50202.log 2023-10-14T01:36:06.869 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49755.log.gz 2023-10-14T01:36:06.869 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53341.log 2023-10-14T01:36:06.870 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42485.log.gz 2023-10-14T01:36:06.870 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40432.log 2023-10-14T01:36:06.870 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50202.log.gz 2023-10-14T01:36:06.870 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54162.log 2023-10-14T01:36:06.871 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53341.log.gz 2023-10-14T01:36:06.871 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55649.log 2023-10-14T01:36:06.871 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2023-10-14T01:36:06.872 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42734.log 2023-10-14T01:36:06.872 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54162.log.gz 2023-10-14T01:36:06.872 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44605.log 2023-10-14T01:36:06.872 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55649.log.gz 2023-10-14T01:36:06.873 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2023-10-14T01:36:06.873 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42734.log.gz 2023-10-14T01:36:06.873 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55384.log 2023-10-14T01:36:06.873 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44605.log.gz 2023-10-14T01:36:06.874 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42512.log 2023-10-14T01:36:06.874 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2023-10-14T01:36:06.874 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54183.log 2023-10-14T01:36:06.875 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55384.log.gz 2023-10-14T01:36:06.875 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.42512.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47872.log 2023-10-14T01:36:06.875 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42512.log.gz 2023-10-14T01:36:06.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49375.log 2023-10-14T01:36:06.876 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54183.log.gz 2023-10-14T01:36:06.876 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41404.log 2023-10-14T01:36:06.876 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47872.log.gz 2023-10-14T01:36:06.876 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46744.log 2023-10-14T01:36:06.877 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49375.log.gz 2023-10-14T01:36:06.877 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2023-10-14T01:36:06.877 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41404.log.gz 2023-10-14T01:36:06.878 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44665.log 2023-10-14T01:36:06.878 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46744.log.gz 2023-10-14T01:36:06.878 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2023-10-14T01:36:06.878 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2023-10-14T01:36:06.879 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56151.log 2023-10-14T01:36:06.879 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44665.log.gz 2023-10-14T01:36:06.879 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2023-10-14T01:36:06.879 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2023-10-14T01:36:06.880 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43254.log 2023-10-14T01:36:06.880 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56151.log.gz 2023-10-14T01:36:06.880 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44270.log 2023-10-14T01:36:06.881 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2023-10-14T01:36:06.881 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55117.log 2023-10-14T01:36:06.881 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43254.log.gz 2023-10-14T01:36:06.881 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44113.log 2023-10-14T01:36:06.882 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44270.log.gz 2023-10-14T01:36:06.882 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56177.log 2023-10-14T01:36:06.882 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55117.log.gz 2023-10-14T01:36:06.882 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2023-10-14T01:36:06.883 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44113.log.gz 2023-10-14T01:36:06.883 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.56177.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53401.log 2023-10-14T01:36:06.883 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56177.log.gz 2023-10-14T01:36:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44049.log 2023-10-14T01:36:06.884 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz 2023-10-14T01:36:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50996.log 2023-10-14T01:36:06.884 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53401.log.gz 2023-10-14T01:36:06.885 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48698.log 2023-10-14T01:36:06.885 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44049.log.gz 2023-10-14T01:36:06.885 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45973.log 2023-10-14T01:36:06.885 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2023-10-14T01:36:06.886 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48698.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-14T01:36:06.886 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48698.log.gz 2023-10-14T01:36:06.886 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43068.log 2023-10-14T01:36:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45973.log.gz 2023-10-14T01:36:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44360.log 2023-10-14T01:36:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46772.log 2023-10-14T01:36:06.887 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43068.log.gz 2023-10-14T01:36:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56821.log 2023-10-14T01:36:06.888 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44360.log.gz 2023-10-14T01:36:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53521.log 2023-10-14T01:36:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46772.log.gz 2023-10-14T01:36:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47621.log 2023-10-14T01:36:06.889 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56821.log.gz 2023-10-14T01:36:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46943.log 2023-10-14T01:36:06.889 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53521.log.gz 2023-10-14T01:36:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.47621.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43202.log 2023-10-14T01:36:06.890 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47621.log.gz 2023-10-14T01:36:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57252.log 2023-10-14T01:36:06.891 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46943.log.gz 2023-10-14T01:36:06.891 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43555.log 2023-10-14T01:36:06.891 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43202.log.gz 2023-10-14T01:36:06.891 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.57252.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46853.log 2023-10-14T01:36:06.892 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57252.log.gz 2023-10-14T01:36:06.892 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56447.log 2023-10-14T01:36:06.892 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43555.log.gz 2023-10-14T01:36:06.892 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49287.log 2023-10-14T01:36:06.893 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46853.log.gz 2023-10-14T01:36:06.893 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45268.log 2023-10-14T01:36:06.893 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56447.log.gz 2023-10-14T01:36:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49287.log.gzgzip 2023-10-14T01:36:06.894 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46052.log 2023-10-14T01:36:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47712.log 2023-10-14T01:36:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45268.log.gz 2023-10-14T01:36:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40540.log 2023-10-14T01:36:06.895 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46052.log.gz 2023-10-14T01:36:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55916.log 2023-10-14T01:36:06.895 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47712.log.gz 2023-10-14T01:36:06.896 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41712.log 2023-10-14T01:36:06.896 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40540.log.gz 2023-10-14T01:36:06.896 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55916.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-14T01:36:06.896 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55916.log.gz 2023-10-14T01:36:06.897 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48146.log 2023-10-14T01:36:06.897 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41712.log.gz 2023-10-14T01:36:06.897 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47841.log 2023-10-14T01:36:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42332.log 2023-10-14T01:36:06.898 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48146.log.gz 2023-10-14T01:36:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42006.log 2023-10-14T01:36:06.898 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47841.log.gz 2023-10-14T01:36:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42706.log 2023-10-14T01:36:06.899 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42332.log.gz 2023-10-14T01:36:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49711.log 2023-10-14T01:36:06.900 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42006.log.gz 2023-10-14T01:36:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43372.log 2023-10-14T01:36:06.900 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42706.log.gz 2023-10-14T01:36:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49235.log 2023-10-14T01:36:06.901 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49711.log.gz 2023-10-14T01:36:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2023-10-14T01:36:06.901 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43372.log.gz 2023-10-14T01:36:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39377.log 2023-10-14T01:36:06.902 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49235.log.gz 2023-10-14T01:36:06.902 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2023-10-14T01:36:06.902 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2023-10-14T01:36:06.903 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54769.log 2023-10-14T01:36:06.903 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39377.log.gz 2023-10-14T01:36:06.903 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.39741.log: --verbose -- /var/log/ceph/ceph-client.admin.46578.log 2023-10-14T01:36:06.903 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2023-10-14T01:36:06.904 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54769.log.gz 2023-10-14T01:36:06.904 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54613.log 2023-10-14T01:36:06.904 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44791.log 2023-10-14T01:36:06.904 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46578.log.gz 2023-10-14T01:36:06.905 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53795.log 2023-10-14T01:36:06.905 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54613.log.gz 2023-10-14T01:36:06.905 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44791.log.gz 2023-10-14T01:36:06.906 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55143.log 2023-10-14T01:36:06.906 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39487.log 2023-10-14T01:36:06.906 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53795.log.gz 2023-10-14T01:36:06.906 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55143.log.gz 2023-10-14T01:36:06.907 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41350.log 2023-10-14T01:36:06.907 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52187.log 2023-10-14T01:36:06.907 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39487.log.gz 2023-10-14T01:36:06.907 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41350.log.gz 2023-10-14T01:36:06.908 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49895.log 2023-10-14T01:36:06.908 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40628.log 2023-10-14T01:36:06.908 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52187.log.gz 2023-10-14T01:36:06.909 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49895.log.gz 2023-10-14T01:36:06.909 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44246.log 2023-10-14T01:36:06.909 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43829.log 2023-10-14T01:36:06.909 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40628.log.gz 2023-10-14T01:36:06.910 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42048.log 2023-10-14T01:36:06.910 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44246.log.gz 2023-10-14T01:36:06.910 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55462.log 2023-10-14T01:36:06.910 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43829.log.gz 2023-10-14T01:36:06.911 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42048.log.gz 2023-10-14T01:36:06.911 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2023-10-14T01:36:06.911 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55462.log.gz 2023-10-14T01:36:06.912 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44390.log 2023-10-14T01:36:06.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2023-10-14T01:36:06.912 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2023-10-14T01:36:06.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42758.log 2023-10-14T01:36:06.913 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44390.log.gz 2023-10-14T01:36:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47681.log 2023-10-14T01:36:06.913 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2023-10-14T01:36:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51633.log 2023-10-14T01:36:06.914 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42758.log.gz 2023-10-14T01:36:06.914 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56847.log 2023-10-14T01:36:06.914 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47681.log.gz 2023-10-14T01:36:06.915 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48498.log 2023-10-14T01:36:06.915 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51633.log.gz 2023-10-14T01:36:06.915 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34670.log 2023-10-14T01:36:06.915 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56847.log.gz 2023-10-14T01:36:06.916 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43969.log 2023-10-14T01:36:06.916 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48498.log.gz 2023-10-14T01:36:06.916 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.34670.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56633.log 2023-10-14T01:36:06.916 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34670.log.gz 2023-10-14T01:36:06.917 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44493.log 2023-10-14T01:36:06.917 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43969.log.gz 2023-10-14T01:36:06.917 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41566.log 2023-10-14T01:36:06.918 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56633.log.gz 2023-10-14T01:36:06.918 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49621.log 2023-10-14T01:36:06.918 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44493.log.gz 2023-10-14T01:36:06.918 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46025.log 2023-10-14T01:36:06.919 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41566.log.gz 2023-10-14T01:36:06.919 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56231.log 2023-10-14T01:36:06.919 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49621.log.gz 2023-10-14T01:36:06.920 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2023-10-14T01:36:06.920 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46025.log.gz 2023-10-14T01:36:06.920 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48035.log 2023-10-14T01:36:06.920 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56231.log.gz 2023-10-14T01:36:06.921 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56099.log 2023-10-14T01:36:06.921 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2023-10-14T01:36:06.921 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39641.log 2023-10-14T01:36:06.921 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48035.log.gz 2023-10-14T01:36:06.922 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53311.log 2023-10-14T01:36:06.922 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56099.log.gz 2023-10-14T01:36:06.922 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43498.log 2023-10-14T01:36:06.923 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39641.log.gz 2023-10-14T01:36:06.923 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50966.log 2023-10-14T01:36:06.923 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53311.log.gz 2023-10-14T01:36:06.923 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57276.log 2023-10-14T01:36:06.924 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43498.log.gz 2023-10-14T01:36:06.924 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40844.log 2023-10-14T01:36:06.924 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50966.log.gz 2023-10-14T01:36:06.924 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2023-10-14T01:36:06.925 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57276.log.gz 2023-10-14T01:36:06.925 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41168.log 2023-10-14T01:36:06.925 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40844.log.gz 2023-10-14T01:36:06.926 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46796.log 2023-10-14T01:36:06.926 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2023-10-14T01:36:06.926 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44635.log 2023-10-14T01:36:06.926 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41168.log.gz 2023-10-14T01:36:06.927 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49956.log 2023-10-14T01:36:06.927 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46796.log.gz 2023-10-14T01:36:06.927 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2023-10-14T01:36:06.928 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44635.log.gz 2023-10-14T01:36:06.928 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51423.log 2023-10-14T01:36:06.928 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49956.log.gz 2023-10-14T01:36:06.928 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40972.log: /var/log/ceph/ceph-client.admin.47096.log 2023-10-14T01:36:06.929 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2023-10-14T01:36:06.929 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51180.log 2023-10-14T01:36:06.929 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51423.log.gz 2023-10-14T01:36:06.929 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51513.log 2023-10-14T01:36:06.930 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47096.log.gz 2023-10-14T01:36:06.930 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46454.log 2023-10-14T01:36:06.930 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51180.log.gz 2023-10-14T01:36:06.931 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43616.log 2023-10-14T01:36:06.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51513.log.gz 2023-10-14T01:36:06.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42632.log 2023-10-14T01:36:06.931 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46454.log.gz 2023-10-14T01:36:06.932 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45020.log 2023-10-14T01:36:06.932 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43616.log.gz 2023-10-14T01:36:06.932 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43646.log 2023-10-14T01:36:06.933 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42632.log.gz 2023-10-14T01:36:06.933 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54847.log 2023-10-14T01:36:06.933 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45020.log.gz 2023-10-14T01:36:06.933 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54456.log 2023-10-14T01:36:06.934 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43646.log.gz 2023-10-14T01:36:06.934 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40989.log 2023-10-14T01:36:06.934 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54847.log.gz 2023-10-14T01:36:06.935 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2023-10-14T01:36:06.935 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54456.log.gz 2023-10-14T01:36:06.935 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48883.log 2023-10-14T01:36:06.935 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40989.log.gz 2023-10-14T01:36:06.936 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34622.log 2023-10-14T01:36:06.936 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2023-10-14T01:36:06.936 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56258.log 2023-10-14T01:36:06.936 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48883.log.gz 2023-10-14T01:36:06.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50875.log 2023-10-14T01:36:06.937 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34622.log.gz 2023-10-14T01:36:06.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56258.log.gz 2023-10-14T01:36:06.938 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40611.log 2023-10-14T01:36:06.938 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2023-10-14T01:36:06.938 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50875.log.gz 2023-10-14T01:36:06.938 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2023-10-14T01:36:06.939 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2023-10-14T01:36:06.939 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53158.log 2023-10-14T01:36:06.939 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2023-10-14T01:36:06.939 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2023-10-14T01:36:06.940 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42878.log 2023-10-14T01:36:06.940 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53491.log 2023-10-14T01:36:06.940 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53158.log.gz 2023-10-14T01:36:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52881.log 2023-10-14T01:36:06.941 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42878.log.gz 2023-10-14T01:36:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53491.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.53765.log -- replaced with /var/log/ceph/ceph-client.admin.53491.log.gz 2023-10-14T01:36:06.941 INFO:teuthology.orchestra.run.smithi081.stderr: 2023-10-14T01:36:06.942 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48794.log 2023-10-14T01:36:06.942 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52881.log.gz 2023-10-14T01:36:06.942 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52125.log 2023-10-14T01:36:06.942 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53765.log.gz 2023-10-14T01:36:06.943 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48852.log 2023-10-14T01:36:06.943 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48794.log.gz 2023-10-14T01:36:06.943 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54256.log 2023-10-14T01:36:06.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52125.log.gz 2023-10-14T01:36:06.944 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.48852.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42542.log 2023-10-14T01:36:06.944 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48852.log.gz 2023-10-14T01:36:06.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54256.log.gz 2023-10-14T01:36:06.945 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40355.log 2023-10-14T01:36:06.945 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2023-10-14T01:36:06.945 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42542.log.gz 2023-10-14T01:36:06.946 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40355.log.gz 2023-10-14T01:36:06.946 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50000.log 2023-10-14T01:36:06.946 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45044.log 2023-10-14T01:36:06.946 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2023-10-14T01:36:06.947 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50028.log 2023-10-14T01:36:06.947 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50000.log.gz 2023-10-14T01:36:06.947 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39567.log 2023-10-14T01:36:06.947 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45044.log.gz 2023-10-14T01:36:06.948 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50028.log.gz 2023-10-14T01:36:06.948 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43891.log 2023-10-14T01:36:06.948 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56926.log 2023-10-14T01:36:06.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39567.log.gz 2023-10-14T01:36:06.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56953.log 2023-10-14T01:36:06.949 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43891.log.gz 2023-10-14T01:36:06.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48209.log 2023-10-14T01:36:06.950 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56926.log.gz 2023-10-14T01:36:06.950 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.56953.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57058.log 2023-10-14T01:36:06.950 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56953.log.gz 2023-10-14T01:36:06.950 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48209.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.48209.log.gz --verbose 2023-10-14T01:36:06.951 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.47762.log 2023-10-14T01:36:06.951 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46699.log 2023-10-14T01:36:06.951 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57058.log.gz 2023-10-14T01:36:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2023-10-14T01:36:06.952 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47762.log.gz 2023-10-14T01:36:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50813.log 2023-10-14T01:36:06.952 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46699.log.gz 2023-10-14T01:36:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53976.log 2023-10-14T01:36:06.953 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2023-10-14T01:36:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52819.log 2023-10-14T01:36:06.953 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50813.log.gz 2023-10-14T01:36:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53976.log.gz 2023-10-14T01:36:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52518.log 2023-10-14T01:36:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50325.log 2023-10-14T01:36:06.955 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52819.log.gz 2023-10-14T01:36:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57006.log 2023-10-14T01:36:06.955 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52518.log.gz 2023-10-14T01:36:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2023-10-14T01:36:06.956 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50325.log.gz 2023-10-14T01:36:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48550.log 2023-10-14T01:36:06.956 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57006.log.gz 2023-10-14T01:36:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55358.log 2023-10-14T01:36:06.957 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2023-10-14T01:36:06.957 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39637.log 2023-10-14T01:36:06.957 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48550.log.gz 2023-10-14T01:36:06.958 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55358.log.gz 2023-10-14T01:36:06.958 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51453.log 2023-10-14T01:36:06.958 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54691.log 2023-10-14T01:36:06.958 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39637.log.gz 2023-10-14T01:36:06.959 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51453.log.gzgzip 2023-10-14T01:36:06.959 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42662.log 2023-10-14T01:36:06.959 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2023-10-14T01:36:06.959 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54691.log.gz 2023-10-14T01:36:06.960 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41796.log 2023-10-14T01:36:06.960 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42662.log.gz 2023-10-14T01:36:06.960 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54546.log 2023-10-14T01:36:06.961 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2023-10-14T01:36:06.961 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2023-10-14T01:36:06.961 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41796.log.gz 2023-10-14T01:36:06.961 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49042.log 2023-10-14T01:36:06.962 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54546.log.gz 2023-10-14T01:36:06.962 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48059.log 2023-10-14T01:36:06.962 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2023-10-14T01:36:06.963 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45513.log 2023-10-14T01:36:06.963 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49042.log.gz 2023-10-14T01:36:06.963 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52218.log 2023-10-14T01:36:06.963 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48059.log.gz 2023-10-14T01:36:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41097.log 2023-10-14T01:36:06.964 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45513.log.gz 2023-10-14T01:36:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53461.log 2023-10-14T01:36:06.964 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52218.log.gz 2023-10-14T01:36:06.965 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45240.log 2023-10-14T01:36:06.965 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41097.log.gz 2023-10-14T01:36:06.965 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54901.log 2023-10-14T01:36:06.965 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53461.log.gz 2023-10-14T01:36:06.966 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44739.log 2023-10-14T01:36:06.966 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45240.log.gz 2023-10-14T01:36:06.966 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.54901.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48822.log 2023-10-14T01:36:06.967 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54901.log.gz 2023-10-14T01:36:06.967 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55942.log 2023-10-14T01:36:06.967 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2023-10-14T01:36:06.967 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56580.log 2023-10-14T01:36:06.968 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48822.log.gz 2023-10-14T01:36:06.968 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42461.log 2023-10-14T01:36:06.968 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55942.log.gz 2023-10-14T01:36:06.968 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51845.log 2023-10-14T01:36:06.969 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56580.log.gz 2023-10-14T01:36:06.969 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50052.log 2023-10-14T01:36:06.969 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42461.log.gz 2023-10-14T01:36:06.970 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51845.log.gz 2023-10-14T01:36:06.970 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44144.log 2023-10-14T01:36:06.970 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50052.log.gz 2023-10-14T01:36:06.970 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51723.log 2023-10-14T01:36:06.971 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53916.log 2023-10-14T01:36:06.971 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44144.log.gz 2023-10-14T01:36:06.971 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2023-10-14T01:36:06.971 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51723.log.gz 2023-10-14T01:36:06.972 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55491.log 2023-10-14T01:36:06.972 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53916.log.gz 2023-10-14T01:36:06.972 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42958.log 2023-10-14T01:36:06.973 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2023-10-14T01:36:06.973 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48577.log 2023-10-14T01:36:06.973 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55491.log.gz 2023-10-14T01:36:06.973 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42280.log 2023-10-14T01:36:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42958.log.gz 2023-10-14T01:36:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2023-10-14T01:36:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48577.log.gz 2023-10-14T01:36:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40827.log 2023-10-14T01:36:06.975 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42280.log.gz 2023-10-14T01:36:06.975 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51150.log 2023-10-14T01:36:06.975 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2023-10-14T01:36:06.976 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46424.log 2023-10-14T01:36:06.976 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40827.log.gz 2023-10-14T01:36:06.976 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48176.log 2023-10-14T01:36:06.976 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51150.log.gz 2023-10-14T01:36:06.977 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50081.log 2023-10-14T01:36:06.977 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46424.log.gz 2023-10-14T01:36:06.977 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49681.log 2023-10-14T01:36:06.977 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48176.log.gz 2023-10-14T01:36:06.978 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.50081.log: --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2023-10-14T01:36:06.978 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50081.log.gz 2023-10-14T01:36:06.978 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43128.log 2023-10-14T01:36:06.979 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49681.log.gz 2023-10-14T01:36:06.979 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-14T01:36:06.979 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2023-10-14T01:36:06.979 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48914.log 2023-10-14T01:36:06.980 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43128.log.gz 2023-10-14T01:36:06.980 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52338.log 2023-10-14T01:36:06.980 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55623.log 2023-10-14T01:36:06.981 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48914.log.gz 2023-10-14T01:36:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54587.log 2023-10-14T01:36:06.981 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52338.log.gz 2023-10-14T01:36:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2023-10-14T01:36:06.982 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55623.log.gz 2023-10-14T01:36:06.982 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49263.log 2023-10-14T01:36:06.982 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54587.log.gz 2023-10-14T01:36:06.982 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54096.log 2023-10-14T01:36:06.983 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2023-10-14T01:36:06.983 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49263.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30937.log 2023-10-14T01:36:06.983 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49263.log.gz 2023-10-14T01:36:06.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40321.log 2023-10-14T01:36:06.984 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54096.log.gz 2023-10-14T01:36:06.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.tmp-client.admin.30937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51363.log 2023-10-14T01:36:06.984 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30937.log.gz 2023-10-14T01:36:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45566.log 2023-10-14T01:36:06.985 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40321.log.gz 2023-10-14T01:36:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57084.log 2023-10-14T01:36:06.986 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51363.log.gz 2023-10-14T01:36:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53551.log 2023-10-14T01:36:06.986 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45566.log.gz 2023-10-14T01:36:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47814.log 2023-10-14T01:36:06.987 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57084.log.gz 2023-10-14T01:36:06.987 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55834.log 2023-10-14T01:36:06.987 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53551.log.gz 2023-10-14T01:36:06.988 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43474.log 2023-10-14T01:36:06.988 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47814.log.gz 2023-10-14T01:36:06.988 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43585.log 2023-10-14T01:36:06.988 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55834.log.gz 2023-10-14T01:36:06.989 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47261.log 2023-10-14T01:36:06.989 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43474.log.gz 2023-10-14T01:36:06.989 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43585.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.43585.log.gz /var/log/ceph/ceph-client.admin.49069.log 2023-10-14T01:36:06.989 INFO:teuthology.orchestra.run.smithi081.stderr: 2023-10-14T01:36:06.990 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55597.log 2023-10-14T01:36:06.990 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47261.log.gz 2023-10-14T01:36:06.990 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50511.log 2023-10-14T01:36:06.991 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49069.log.gz 2023-10-14T01:36:06.991 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49533.log 2023-10-14T01:36:06.991 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55597.log.gz 2023-10-14T01:36:06.991 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48668.log 2023-10-14T01:36:06.992 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50511.log.gz 2023-10-14T01:36:06.992 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50298.log 2023-10-14T01:36:06.992 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49533.log.gz 2023-10-14T01:36:06.992 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50601.log 2023-10-14T01:36:06.993 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48668.log.gz 2023-10-14T01:36:06.993 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50691.log 2023-10-14T01:36:06.993 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50298.log.gz 2023-10-14T01:36:06.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43011.log 2023-10-14T01:36:06.994 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50601.log.gz 2023-10-14T01:36:06.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50451.log 2023-10-14T01:36:06.994 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50691.log.gz 2023-10-14T01:36:06.995 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46115.log 2023-10-14T01:36:06.995 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43011.log.gz 2023-10-14T01:36:06.995 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50451.log.gz 2023-10-14T01:36:06.996 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2023-10-14T01:36:06.996 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52912.log 2023-10-14T01:36:06.996 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46115.log.gz 2023-10-14T01:36:06.996 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.40304.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54347.log 2023-10-14T01:36:06.997 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2023-10-14T01:36:06.997 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44939.log 2023-10-14T01:36:06.997 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52912.log.gz 2023-10-14T01:36:06.997 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51573.log 2023-10-14T01:36:06.998 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54347.log.gz 2023-10-14T01:36:06.998 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54821.log 2023-10-14T01:36:06.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44939.log.gz 2023-10-14T01:36:06.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40665.log 2023-10-14T01:36:06.999 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51573.log.gz 2023-10-14T01:36:06.999 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56369.log 2023-10-14T01:36:06.999 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54821.log.gz 2023-10-14T01:36:07.000 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51240.log 2023-10-14T01:36:07.000 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40665.log.gz 2023-10-14T01:36:07.000 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41669.log 2023-10-14T01:36:07.000 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56369.log.gz 2023-10-14T01:36:07.001 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56421.log 2023-10-14T01:36:07.001 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51240.log.gz 2023-10-14T01:36:07.001 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53096.log 2023-10-14T01:36:07.001 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2023-10-14T01:36:07.002 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46303.log 2023-10-14T01:36:07.002 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56421.log.gz 2023-10-14T01:36:07.002 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56686.log 2023-10-14T01:36:07.003 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53096.log.gz 2023-10-14T01:36:07.003 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2023-10-14T01:36:07.003 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46303.log.gz 2023-10-14T01:36:07.003 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39403.log 2023-10-14T01:36:07.004 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56686.log.gz 2023-10-14T01:36:07.004 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45656.log 2023-10-14T01:36:07.004 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2023-10-14T01:36:07.004 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54142.log 2023-10-14T01:36:07.005 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39403.log.gz 2023-10-14T01:36:07.005 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45656.log.gz 2023-10-14T01:36:07.005 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50661.log 2023-10-14T01:36:07.005 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.54142.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43686.log 2023-10-14T01:36:07.006 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54142.log.gz 2023-10-14T01:36:07.006 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44174.log 2023-10-14T01:36:07.006 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50661.log.gz 2023-10-14T01:36:07.007 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43686.log.gz 2023-10-14T01:36:07.007 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47188.log 2023-10-14T01:36:07.007 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.44174.log: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-14T01:36:07.007 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44174.log.gz 2023-10-14T01:36:07.008 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47188.log: -- /var/log/ceph/ceph-client.admin.53065.log 2023-10-14T01:36:07.008 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47188.log.gz 2023-10-14T01:36:07.008 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41754.log 2023-10-14T01:36:07.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47902.log 2023-10-14T01:36:07.009 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53065.log.gz 2023-10-14T01:36:07.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2023-10-14T01:36:07.009 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41754.log.gz 2023-10-14T01:36:07.010 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39818.log 2023-10-14T01:36:07.010 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47902.log.gz 2023-10-14T01:36:07.010 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55436.log 2023-10-14T01:36:07.010 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2023-10-14T01:36:07.011 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46147.log 2023-10-14T01:36:07.011 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39818.log.gz 2023-10-14T01:36:07.011 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34705.log 2023-10-14T01:36:07.012 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55436.log.gz 2023-10-14T01:36:07.012 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41367.log 2023-10-14T01:36:07.012 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46147.log.gz 2023-10-14T01:36:07.012 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43402.log 2023-10-14T01:36:07.013 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34705.log.gz 2023-10-14T01:36:07.013 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50783.log 2023-10-14T01:36:07.013 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41367.log.gz 2023-10-14T01:36:07.013 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43402.log.gz 2023-10-14T01:36:07.014 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55037.log 2023-10-14T01:36:07.014 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55306.log 2023-10-14T01:36:07.014 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50783.log.gz 2023-10-14T01:36:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47039.log 2023-10-14T01:36:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55037.log.gz 2023-10-14T01:36:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55306.log.gz 2023-10-14T01:36:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47591.log 2023-10-14T01:36:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2023-10-14T01:36:07.016 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47039.log.gz 2023-10-14T01:36:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52368.log 2023-10-14T01:36:07.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47591.log.gz 2023-10-14T01:36:07.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gzgzip 2023-10-14T01:36:07.017 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T01:36:07.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48946.log 2023-10-14T01:36:07.018 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52368.log.gz 2023-10-14T01:36:07.018 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51210.log 2023-10-14T01:36:07.018 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48946.log.gz 2023-10-14T01:36:07.018 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2023-10-14T01:36:07.019 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2023-10-14T01:36:07.019 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51210.log.gz 2023-10-14T01:36:07.019 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52698.log 2023-10-14T01:36:07.019 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2023-10-14T01:36:07.020 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41080.log 2023-10-14T01:36:07.020 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2023-10-14T01:36:07.020 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52698.log.gz 2023-10-14T01:36:07.021 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48637.log 2023-10-14T01:36:07.021 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49437.log 2023-10-14T01:36:07.021 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41080.log.gz 2023-10-14T01:36:07.021 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52094.log 2023-10-14T01:36:07.022 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48637.log.gz 2023-10-14T01:36:07.022 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51059.log 2023-10-14T01:36:07.022 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49437.log.gz 2023-10-14T01:36:07.022 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2023-10-14T01:36:07.023 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52094.log.gz 2023-10-14T01:36:07.023 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2023-10-14T01:36:07.023 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51059.log.gz 2023-10-14T01:36:07.024 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52033.log 2023-10-14T01:36:07.024 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2023-10-14T01:36:07.024 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2023-10-14T01:36:07.024 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2023-10-14T01:36:07.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46499.log 2023-10-14T01:36:07.025 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52033.log.gz 2023-10-14T01:36:07.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2023-10-14T01:36:07.025 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2023-10-14T01:36:07.026 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43098.log 2023-10-14T01:36:07.026 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46499.log.gz 2023-10-14T01:36:07.026 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47233.log 2023-10-14T01:36:07.027 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2023-10-14T01:36:07.027 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45538.log 2023-10-14T01:36:07.027 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43098.log.gz 2023-10-14T01:36:07.027 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.47233.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2023-10-14T01:36:07.028 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47233.log.gz 2023-10-14T01:36:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47314.log 2023-10-14T01:36:07.028 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45538.log.gz 2023-10-14T01:36:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2023-10-14T01:36:07.029 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.56073.log 2023-10-14T01:36:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47314.log.gz 2023-10-14T01:36:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49835.log 2023-10-14T01:36:07.030 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54875.log 2023-10-14T01:36:07.030 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56073.log.gz 2023-10-14T01:36:07.030 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45485.log 2023-10-14T01:36:07.030 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49835.log.gz 2023-10-14T01:36:07.031 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40503.log 2023-10-14T01:36:07.031 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54875.log.gz 2023-10-14T01:36:07.031 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46987.log 2023-10-14T01:36:07.031 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45485.log.gz 2023-10-14T01:36:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40503.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.45758.log 2023-10-14T01:36:07.032 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40503.log.gz 2023-10-14T01:36:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41964.log 2023-10-14T01:36:07.033 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46987.log.gz 2023-10-14T01:36:07.033 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49651.log 2023-10-14T01:36:07.033 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45758.log.gz 2023-10-14T01:36:07.033 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41964.log.gz 2023-10-14T01:36:07.034 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2023-10-14T01:36:07.034 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2023-10-14T01:36:07.034 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49651.log.gz 2023-10-14T01:36:07.035 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55995.log 2023-10-14T01:36:07.035 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz 2023-10-14T01:36:07.035 INFO:teuthology.orchestra.run.smithi081.stderr: 89.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T01:36:07.035 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39263.log 2023-10-14T01:36:07.036 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2023-10-14T01:36:07.036 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48607.log 2023-10-14T01:36:07.036 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55995.log.gz 2023-10-14T01:36:07.036 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51029.log 2023-10-14T01:36:07.037 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39263.log.gz 2023-10-14T01:36:07.037 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56713.log 2023-10-14T01:36:07.037 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48607.log.gz 2023-10-14T01:36:07.037 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2023-10-14T01:36:07.038 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51029.log.gz 2023-10-14T01:36:07.038 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49865.log 2023-10-14T01:36:07.038 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56713.log.gz 2023-10-14T01:36:07.039 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45730.log 2023-10-14T01:36:07.039 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2023-10-14T01:36:07.039 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2023-10-14T01:36:07.039 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49865.log.gz 2023-10-14T01:36:07.040 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52788.log 2023-10-14T01:36:07.040 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45730.log.gz 2023-10-14T01:36:07.040 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55890.log 2023-10-14T01:36:07.040 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2023-10-14T01:36:07.041 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50142.log 2023-10-14T01:36:07.041 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52788.log.gz 2023-10-14T01:36:07.041 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55410.log 2023-10-14T01:36:07.042 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55890.log.gz 2023-10-14T01:36:07.042 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46274.log 2023-10-14T01:36:07.042 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50142.log.gz 2023-10-14T01:36:07.042 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42986.log 2023-10-14T01:36:07.043 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55410.log.gz 2023-10-14T01:36:07.043 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54437.log 2023-10-14T01:36:07.043 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46274.log.gz 2023-10-14T01:36:07.043 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46551.log 2023-10-14T01:36:07.044 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42986.log.gz 2023-10-14T01:36:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42816.log 2023-10-14T01:36:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54437.log.gz 2023-10-14T01:36:07.045 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54365.log 2023-10-14T01:36:07.045 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46551.log.gz 2023-10-14T01:36:07.045 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52578.log 2023-10-14T01:36:07.045 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42816.log.gz 2023-10-14T01:36:07.046 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42132.log 2023-10-14T01:36:07.046 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54365.log.gz 2023-10-14T01:36:07.046 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56047.log 2023-10-14T01:36:07.047 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52578.log.gz 2023-10-14T01:36:07.047 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41880.log 2023-10-14T01:36:07.047 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42132.log.gz 2023-10-14T01:36:07.047 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54528.log/var/log/ceph/ceph-client.admin.56047.log: 2023-10-14T01:36:07.048 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56047.log.gz 2023-10-14T01:36:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50358.log 2023-10-14T01:36:07.048 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41880.log.gz 2023-10-14T01:36:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39455.log 2023-10-14T01:36:07.049 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54528.log.gz 2023-10-14T01:36:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2023-10-14T01:36:07.049 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50358.log.gz 2023-10-14T01:36:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56343.log 2023-10-14T01:36:07.050 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39455.log.gz 2023-10-14T01:36:07.050 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55280.log 2023-10-14T01:36:07.050 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2023-10-14T01:36:07.051 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53704.log 2023-10-14T01:36:07.051 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56343.log.gz 2023-10-14T01:36:07.051 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50541.log 2023-10-14T01:36:07.051 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55280.log.gz 2023-10-14T01:36:07.052 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49560.log 2023-10-14T01:36:07.052 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53704.log.gz 2023-10-14T01:36:07.052 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48393.log 2023-10-14T01:36:07.052 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50541.log.gz 2023-10-14T01:36:07.053 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52758.log 2023-10-14T01:36:07.053 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49560.log.gz 2023-10-14T01:36:07.053 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43342.log 2023-10-14T01:36:07.054 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48393.log.gz 2023-10-14T01:36:07.054 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.52758.log: --verbose -- /var/log/ceph/ceph-client.admin.53281.log 2023-10-14T01:36:07.054 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52758.log.gz 2023-10-14T01:36:07.054 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44767.log 2023-10-14T01:36:07.055 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43342.log.gz 2023-10-14T01:36:07.055 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56659.log 2023-10-14T01:36:07.055 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53281.log.gz 2023-10-14T01:36:07.056 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54743.log 2023-10-14T01:36:07.056 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44767.log.gz 2023-10-14T01:36:07.056 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.56659.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53220.log 2023-10-14T01:36:07.056 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56659.log.gz 2023-10-14T01:36:07.057 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54743.log: /var/log/ceph/ceph-client.admin.56899.log 2023-10-14T01:36:07.057 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54743.log.gz 2023-10-14T01:36:07.057 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40648.log 2023-10-14T01:36:07.057 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53220.log.gz 2023-10-14T01:36:07.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2023-10-14T01:36:07.058 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56899.log.gz 2023-10-14T01:36:07.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53127.log 2023-10-14T01:36:07.059 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40648.log.gz 2023-10-14T01:36:07.059 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.39559.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52974.log 2023-10-14T01:36:07.059 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2023-10-14T01:36:07.059 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55727.log 2023-10-14T01:36:07.060 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53127.log.gz 2023-10-14T01:36:07.060 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43446.log 2023-10-14T01:36:07.060 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52974.log.gz 2023-10-14T01:36:07.060 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2023-10-14T01:36:07.061 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55727.log.gz 2023-10-14T01:36:07.061 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.43446.log: --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T01:36:07.061 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43446.log.gz 2023-10-14T01:36:07.062 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48253.log 2023-10-14T01:36:07.062 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2023-10-14T01:36:07.062 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46608.log 2023-10-14T01:36:07.062 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.48253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44848.log 2023-10-14T01:36:07.063 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48253.log.gz 2023-10-14T01:36:07.063 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51333.log 2023-10-14T01:36:07.063 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46608.log.gz 2023-10-14T01:36:07.064 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2023-10-14T01:36:07.064 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44848.log.gz 2023-10-14T01:36:07.064 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46883.log 2023-10-14T01:36:07.064 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51333.log.gz 2023-10-14T01:36:07.065 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46085.log 2023-10-14T01:36:07.065 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2023-10-14T01:36:07.065 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54329.log 2023-10-14T01:36:07.066 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46883.log.gz 2023-10-14T01:36:07.066 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54927.log 2023-10-14T01:36:07.066 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46085.log.gz 2023-10-14T01:36:07.066 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47508.log 2023-10-14T01:36:07.067 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54329.log.gz 2023-10-14T01:36:07.067 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56125.log 2023-10-14T01:36:07.067 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54927.log.gz 2023-10-14T01:36:07.067 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51543.log 2023-10-14T01:36:07.068 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47508.log.gz 2023-10-14T01:36:07.068 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53671.log 2023-10-14T01:36:07.068 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56125.log.gz 2023-10-14T01:36:07.069 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43720.log 2023-10-14T01:36:07.069 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51543.log.gz 2023-10-14T01:36:07.069 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44909.log 2023-10-14T01:36:07.069 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53671.log.gz 2023-10-14T01:36:07.070 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43230.log 2023-10-14T01:36:07.070 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43720.log.gz 2023-10-14T01:36:07.070 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49129.log 2023-10-14T01:36:07.070 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44909.log.gz 2023-10-14T01:36:07.071 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55864.log 2023-10-14T01:36:07.071 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43230.log.gz 2023-10-14T01:36:07.071 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45411.log 2023-10-14T01:36:07.072 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49129.log.gz 2023-10-14T01:36:07.073 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41188.log 2023-10-14T01:36:07.073 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55864.log.gz 2023-10-14T01:36:07.074 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56021.log 2023-10-14T01:36:07.074 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45411.log.gz 2023-10-14T01:36:07.074 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42389.log 2023-10-14T01:36:07.075 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2023-10-14T01:36:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47015.log 2023-10-14T01:36:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56021.log.gz 2023-10-14T01:36:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48116.log 2023-10-14T01:36:07.076 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42389.log.gz 2023-10-14T01:36:07.076 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50905.log 2023-10-14T01:36:07.076 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47015.log.gz 2023-10-14T01:36:07.077 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42785.log 2023-10-14T01:36:07.077 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48116.log.gz 2023-10-14T01:36:07.077 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2023-10-14T01:36:07.077 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50905.log.gz 2023-10-14T01:36:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2023-10-14T01:36:07.078 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42785.log.gz 2023-10-14T01:36:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42846.log 2023-10-14T01:36:07.079 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2023-10-14T01:36:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51967.log 2023-10-14T01:36:07.079 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2023-10-14T01:36:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44545.log 2023-10-14T01:36:07.080 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42846.log.gz 2023-10-14T01:36:07.080 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49926.log 2023-10-14T01:36:07.080 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51967.log.gz 2023-10-14T01:36:07.080 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54639.log 2023-10-14T01:36:07.081 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44545.log.gz 2023-10-14T01:36:07.081 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41006.log 2023-10-14T01:36:07.081 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49926.log.gz 2023-10-14T01:36:07.082 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54639.log: /var/log/ceph/ceph-client.admin.45869.log 2023-10-14T01:36:07.082 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54639.log.gz 2023-10-14T01:36:07.082 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53825.log 2023-10-14T01:36:07.082 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41006.log.gz 2023-10-14T01:36:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45782.log 2023-10-14T01:36:07.083 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45869.log.gz 2023-10-14T01:36:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47287.log 2023-10-14T01:36:07.083 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53825.log.gz 2023-10-14T01:36:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49405.log 2023-10-14T01:36:07.084 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45782.log.gz 2023-10-14T01:36:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52608.log 2023-10-14T01:36:07.085 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47287.log.gz 2023-10-14T01:36:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48306.log 2023-10-14T01:36:07.085 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49405.log.gz 2023-10-14T01:36:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53946.log 2023-10-14T01:36:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52608.log.gz 2023-10-14T01:36:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51907.log 2023-10-14T01:36:07.086 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48306.log.gz 2023-10-14T01:36:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45899.log 2023-10-14T01:36:07.087 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53946.log.gz 2023-10-14T01:36:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50481.log 2023-10-14T01:36:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51907.log.gz 2023-10-14T01:36:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46218.log 2023-10-14T01:36:07.088 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45899.log.gz 2023-10-14T01:36:07.088 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56766.log 2023-10-14T01:36:07.088 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50481.log.gz 2023-10-14T01:36:07.089 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48424.log 2023-10-14T01:36:07.089 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46218.log.gz 2023-10-14T01:36:07.089 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.56766.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47651.log 2023-10-14T01:36:07.089 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56766.log.gz 2023-10-14T01:36:07.090 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42602.log 2023-10-14T01:36:07.090 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48424.log.gz 2023-10-14T01:36:07.090 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.47651.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42359.log 2023-10-14T01:36:07.091 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47651.log.gz 2023-10-14T01:36:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52728.log 2023-10-14T01:36:07.091 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42602.log.gz 2023-10-14T01:36:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47561.log 2023-10-14T01:36:07.092 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42359.log.gz 2023-10-14T01:36:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47932.log 2023-10-14T01:36:07.092 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52728.log.gz 2023-10-14T01:36:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52849.log 2023-10-14T01:36:07.093 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47561.log.gz 2023-10-14T01:36:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47932.log.gz 2023-10-14T01:36:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2023-10-14T01:36:07.094 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52849.log.gz 2023-10-14T01:36:07.094 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40338.log 2023-10-14T01:36:07.094 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55968.log 2023-10-14T01:36:07.095 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2023-10-14T01:36:07.095 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52944.log 2023-10-14T01:36:07.095 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40338.log.gz 2023-10-14T01:36:07.095 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53734.log 2023-10-14T01:36:07.096 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55968.log.gz 2023-10-14T01:36:07.096 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2023-10-14T01:36:07.096 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52944.log.gz 2023-10-14T01:36:07.096 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45320.log 2023-10-14T01:36:07.097 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53734.log.gz 2023-10-14T01:36:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.40952.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52668.log 2023-10-14T01:36:07.097 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2023-10-14T01:36:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54274.log 2023-10-14T01:36:07.098 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45320.log.gz 2023-10-14T01:36:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2023-10-14T01:36:07.098 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52668.log.gz 2023-10-14T01:36:07.099 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53189.log 2023-10-14T01:36:07.099 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54274.log.gz 2023-10-14T01:36:07.099 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51393.log 2023-10-14T01:36:07.099 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2023-10-14T01:36:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52428.log 2023-10-14T01:36:07.100 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53189.log.gz 2023-10-14T01:36:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47476.log 2023-10-14T01:36:07.101 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51393.log.gz 2023-10-14T01:36:07.101 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46363.log 2023-10-14T01:36:07.101 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52428.log.gz 2023-10-14T01:36:07.101 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.47476.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34528.log 2023-10-14T01:36:07.102 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47476.log.gz 2023-10-14T01:36:07.102 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56314.log 2023-10-14T01:36:07.102 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46363.log.gz 2023-10-14T01:36:07.102 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43997.log 2023-10-14T01:36:07.103 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34528.log.gz 2023-10-14T01:36:07.103 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55517.log 2023-10-14T01:36:07.103 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56314.log.gz 2023-10-14T01:36:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49161.log 2023-10-14T01:36:07.104 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43997.log.gz 2023-10-14T01:36:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52308.log 2023-10-14T01:36:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55517.log.gz 2023-10-14T01:36:07.105 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51815.log/var/log/ceph/ceph-client.admin.49161.log: 2023-10-14T01:36:07.105 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49161.log.gz 2023-10-14T01:36:07.105 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2023-10-14T01:36:07.105 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52308.log.gz 2023-10-14T01:36:07.106 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52278.log 2023-10-14T01:36:07.106 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51815.log.gz 2023-10-14T01:36:07.106 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57164.log 2023-10-14T01:36:07.107 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2023-10-14T01:36:07.107 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45350.log 2023-10-14T01:36:07.107 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52278.log.gz 2023-10-14T01:36:07.107 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43799.log 2023-10-14T01:36:07.108 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57164.log.gz 2023-10-14T01:36:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49481.log 2023-10-14T01:36:07.108 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45350.log.gz 2023-10-14T01:36:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2023-10-14T01:36:07.109 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43799.log.gz 2023-10-14T01:36:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39715.log 2023-10-14T01:36:07.109 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49481.log.gz 2023-10-14T01:36:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55754.log 2023-10-14T01:36:07.110 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2023-10-14T01:36:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57138.log 2023-10-14T01:36:07.110 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz 2023-10-14T01:36:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56285.log 2023-10-14T01:36:07.111 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55754.log.gz 2023-10-14T01:36:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54419.log 2023-10-14T01:36:07.112 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57138.log.gz 2023-10-14T01:36:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41470.log 2023-10-14T01:36:07.112 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56285.log.gz 2023-10-14T01:36:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48454.log 2023-10-14T01:36:07.113 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54419.log.gz 2023-10-14T01:36:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48363.log 2023-10-14T01:36:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41470.log.gz 2023-10-14T01:36:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43525.log 2023-10-14T01:36:07.114 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48454.log.gz 2023-10-14T01:36:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56554.log 2023-10-14T01:36:07.114 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48363.log.gz 2023-10-14T01:36:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2023-10-14T01:36:07.115 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2023-10-14T01:36:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45839.log 2023-10-14T01:36:07.115 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56554.log.gz 2023-10-14T01:36:07.116 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2023-10-14T01:36:07.116 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2023-10-14T01:36:07.116 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2023-10-14T01:36:07.117 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45839.log.gz 2023-10-14T01:36:07.117 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51663.log 2023-10-14T01:36:07.117 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2023-10-14T01:36:07.117 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55065.log 2023-10-14T01:36:07.118 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2023-10-14T01:36:07.118 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.51663.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2023-10-14T01:36:07.118 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51663.log.gz 2023-10-14T01:36:07.118 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2023-10-14T01:36:07.119 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55065.log.gz 2023-10-14T01:36:07.119 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54036.log 2023-10-14T01:36:07.119 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2023-10-14T01:36:07.120 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.41932.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2023-10-14T01:36:07.120 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2023-10-14T01:36:07.120 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2023-10-14T01:36:07.120 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54036.log.gz 2023-10-14T01:36:07.121 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56204.log 2023-10-14T01:36:07.121 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2023-10-14T01:36:07.121 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50388.log 2023-10-14T01:36:07.122 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2023-10-14T01:36:07.122 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44327.log 2023-10-14T01:36:07.122 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56204.log.gz 2023-10-14T01:36:07.122 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47963.log 2023-10-14T01:36:07.123 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50388.log.gz 2023-10-14T01:36:07.123 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53371.log 2023-10-14T01:36:07.123 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44327.log.gz 2023-10-14T01:36:07.123 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46001.log 2023-10-14T01:36:07.124 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47963.log.gz 2023-10-14T01:36:07.124 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48742.log 2023-10-14T01:36:07.124 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53371.log.gz 2023-10-14T01:36:07.125 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50421.log 2023-10-14T01:36:07.125 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46001.log.gz 2023-10-14T01:36:07.125 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46669.log 2023-10-14T01:36:07.125 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48742.log.gz 2023-10-14T01:36:07.126 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2023-10-14T01:36:07.126 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50421.log.gz 2023-10-14T01:36:07.126 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54401.log 2023-10-14T01:36:07.127 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46669.log.gz 2023-10-14T01:36:07.127 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50246.log 2023-10-14T01:36:07.127 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2023-10-14T01:36:07.127 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51090.log 2023-10-14T01:36:07.128 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54401.log.gz 2023-10-14T01:36:07.128 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55543.log 2023-10-14T01:36:07.128 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50246.log.gz 2023-10-14T01:36:07.128 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49191.log 2023-10-14T01:36:07.129 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51090.log.gz 2023-10-14T01:36:07.129 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51937.log 2023-10-14T01:36:07.129 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55543.log.gz 2023-10-14T01:36:07.129 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49191.log: /var/log/ceph/ceph-client.admin.47157.log 2023-10-14T01:36:07.130 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49191.log.gz 2023-10-14T01:36:07.130 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51876.log 2023-10-14T01:36:07.130 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51937.log.gz 2023-10-14T01:36:07.131 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54492.log 2023-10-14T01:36:07.131 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47157.log.gz 2023-10-14T01:36:07.131 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55226.log 2023-10-14T01:36:07.131 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51876.log.gz 2023-10-14T01:36:07.132 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53611.log 2023-10-14T01:36:07.132 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54492.log.gz 2023-10-14T01:36:07.132 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41151.log 2023-10-14T01:36:07.133 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55226.log.gz 2023-10-14T01:36:07.133 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42226.log 2023-10-14T01:36:07.133 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53611.log.gz 2023-10-14T01:36:07.133 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41534.log 2023-10-14T01:36:07.134 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41151.log.gz 2023-10-14T01:36:07.134 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47405.log 2023-10-14T01:36:07.134 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42226.log.gz 2023-10-14T01:36:07.134 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 2023-10-14T01:36:07.135 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41534.log.gz 2023-10-14T01:36:07.135 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57032.log 2023-10-14T01:36:07.135 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47405.log.gz 2023-10-14T01:36:07.136 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.40412.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56792.log 2023-10-14T01:36:07.136 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gz 2023-10-14T01:36:07.136 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42182.log 2023-10-14T01:36:07.136 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57032.log.gz 2023-10-14T01:36:07.137 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2023-10-14T01:36:07.137 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56792.log.gz 2023-10-14T01:36:07.137 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50845.log 2023-10-14T01:36:07.137 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42182.log.gz 2023-10-14T01:36:07.138 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49018.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41384.log 2023-10-14T01:36:07.138 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz 2023-10-14T01:36:07.138 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45929.log 2023-10-14T01:36:07.139 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50845.log.gz 2023-10-14T01:36:07.139 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45195.log 2023-10-14T01:36:07.139 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41384.log.gz 2023-10-14T01:36:07.139 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51303.log 2023-10-14T01:36:07.140 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45929.log.gz 2023-10-14T01:36:07.140 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43158.log 2023-10-14T01:36:07.140 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45195.log.gz 2023-10-14T01:36:07.141 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.51303.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2023-10-14T01:36:07.141 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51303.log.gz 2023-10-14T01:36:07.141 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57224.log 2023-10-14T01:36:07.141 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43158.log.gz 2023-10-14T01:36:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56873.log 2023-10-14T01:36:07.142 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2023-10-14T01:36:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.57224.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44218.log 2023-10-14T01:36:07.142 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57224.log.gz 2023-10-14T01:36:07.143 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55169.log 2023-10-14T01:36:07.143 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56873.log.gz 2023-10-14T01:36:07.143 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42433.log 2023-10-14T01:36:07.144 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44218.log.gz 2023-10-14T01:36:07.144 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46394.log 2023-10-14T01:36:07.144 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2023-10-14T01:36:07.144 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42433.log.gz 2023-10-14T01:36:07.145 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46394.log.gz 2023-10-14T01:36:07.145 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46638.log 2023-10-14T01:36:07.145 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55010.log 2023-10-14T01:36:07.145 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2023-10-14T01:36:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50112.log 2023-10-14T01:36:07.146 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46638.log.gz 2023-10-14T01:36:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41259.log 2023-10-14T01:36:07.147 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55010.log.gz 2023-10-14T01:36:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51755.log 2023-10-14T01:36:07.147 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50112.log.gz 2023-10-14T01:36:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54955.log 2023-10-14T01:36:07.148 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41259.log.gz 2023-10-14T01:36:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55701.log 2023-10-14T01:36:07.148 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51755.log.gz 2023-10-14T01:36:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52548.log 2023-10-14T01:36:07.149 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54955.log.gz 2023-10-14T01:36:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41205.log 2023-10-14T01:36:07.149 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55701.log.gz 2023-10-14T01:36:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51999.log 2023-10-14T01:36:07.150 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52548.log.gz 2023-10-14T01:36:07.150 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57110.log 2023-10-14T01:36:07.150 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41205.log.gz 2023-10-14T01:36:07.151 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48526.log 2023-10-14T01:36:07.151 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51999.log.gz 2023-10-14T01:36:07.151 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2023-10-14T01:36:07.152 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57110.log.gz 2023-10-14T01:36:07.152 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2023-10-14T01:36:07.152 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48526.log.gz 2023-10-14T01:36:07.152 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47375.log 2023-10-14T01:36:07.153 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2023-10-14T01:36:07.153 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56979.log 2023-10-14T01:36:07.153 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2023-10-14T01:36:07.153 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50752.log/var/log/ceph/ceph-client.admin.47375.log: 2023-10-14T01:36:07.154 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47375.log.gz 2023-10-14T01:36:07.154 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2023-10-14T01:36:07.154 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56979.log.gz 2023-10-14T01:36:07.155 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48086.log 2023-10-14T01:36:07.155 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50752.log.gz 2023-10-14T01:36:07.155 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2023-10-14T01:36:07.155 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52458.log 2023-10-14T01:36:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51120.log 2023-10-14T01:36:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48086.log.gz 2023-10-14T01:36:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56501.log 2023-10-14T01:36:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52458.log.gz 2023-10-14T01:36:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44297.log 2023-10-14T01:36:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51120.log.gz 2023-10-14T01:36:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46250.log 2023-10-14T01:36:07.158 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56501.log.gz 2023-10-14T01:36:07.158 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42572.log 2023-10-14T01:36:07.158 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44297.log.gz 2023-10-14T01:36:07.158 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46250.log.gz 2023-10-14T01:36:07.159 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44420.log 2023-10-14T01:36:07.159 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42572.log.gz 2023-10-14T01:36:07.159 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44420.log.gz 2023-10-14T01:36:07.160 INFO:teuthology.orchestra.run.smithi081.stderr: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.55169.log.gz 2023-10-14T01:36:07.160 INFO:teuthology.orchestra.run.smithi081.stderr: 90.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T01:36:08.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-14T01:36:19.693 INFO:teuthology.orchestra.run.smithi081.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-14T01:36:26.090 INFO:teuthology.orchestra.run.smithi081.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-14T02:11:44.940 INFO:teuthology.orchestra.run.smithi081.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-14T02:23:44.675 INFO:teuthology.orchestra.run.smithi081.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-14T02:23:52.546 INFO:teuthology.orchestra.run.smithi081.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-14T02:23:52.548 INFO:teuthology.orchestra.run.smithi081.stderr: 2023-10-14T02:23:52.548 INFO:teuthology.orchestra.run.smithi081.stderr:real 47m45.937s 2023-10-14T02:23:52.549 INFO:teuthology.orchestra.run.smithi081.stderr:user 45m11.747s 2023-10-14T02:23:52.549 INFO:teuthology.orchestra.run.smithi081.stderr:sys 2m23.422s 2023-10-14T02:23:52.549 INFO:tasks.ceph:Archiving logs... 2023-10-14T02:23:52.551 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279/remote/smithi081/log 2023-10-14T02:23:52.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T02:25:56.790 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T02:25:56.802 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-14T02:25:56.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T02:25:56.846 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-14T02:25:56.857 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-14T02:25:56.857 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T02:25:56.954 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2023-10-14T02:25:56.955 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2023-10-14T02:25:56.955 INFO:teuthology.orchestra.run.smithi081.stdout:*hv01.front.sepi 44.4.53.2 2 u 103 256 377 0.194 -0.531 0.529 2023-10-14T02:25:56.956 INFO:teuthology.orchestra.run.smithi081.stdout:+hv02.front.sepi 63.231.80.2 3 u 206 256 377 0.200 -1.043 0.344 2023-10-14T02:25:56.956 INFO:teuthology.orchestra.run.smithi081.stdout:+hv03.front.sepi 74.6.168.72 3 u 112 256 377 0.181 -0.796 0.182 2023-10-14T02:25:56.957 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-14T02:25:56.959 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-14T02:25:56.968 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-14T02:25:56.969 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-14T02:25:56.978 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-14T02:25:56.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-14T02:25:57.002 INFO:teuthology.task.internal:Duration was 9051.540663 seconds 2023-10-14T02:25:57.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-14T02:25:57.015 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-14T02:25:57.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T02:25:57.389 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-14T02:25:57.389 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2023-10-14T02:25:57.390 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T02:25:57.447 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-14T02:25:57.448 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T02:25:57.561 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-14T02:25:57.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T02:25:58.068 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-14T02:25:58.079 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-14T02:25:58.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T02:25:58.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-14T02:25:58.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T02:25:58.176 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2023-10-14T02:25:58.207 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T02:25:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T02:25:58.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-14T02:25:58.268 INFO:teuthology.task.internal:Transferring archived files... 2023-10-14T02:25:58.270 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279/remote/smithi081 2023-10-14T02:25:58.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T02:25:58.327 INFO:teuthology.task.internal:Removing archive directory... 2023-10-14T02:25:58.327 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T02:25:58.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-14T02:25:58.381 INFO:teuthology.task.internal:Not uploading archives. 2023-10-14T02:25:58.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-14T02:25:58.391 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-14T02:25:58.392 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T02:25:58.433 INFO:teuthology.orchestra.run.smithi081.stdout: 655475 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 14 02:25 /home/ubuntu/cephtest 2023-10-14T02:25:58.434 INFO:teuthology.orchestra.run.smithi081.stdout: 673507 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 14 00:05 /home/ubuntu/cephtest/mnt.0 2023-10-14T02:25:58.434 INFO:teuthology.orchestra.run.smithi081.stdout: 673933 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 14 00:07 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-14T02:25:58.435 INFO:teuthology.orchestra.run.smithi081.stdout: 802642 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 14 00:07 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-14T02:25:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T02:25:58.436 INFO:teuthology.orchestra.run.smithi081.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-14T02:25:58.437 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-14T02:25:58.446 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-14T02:25:58.459 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-14T02:25:58.472 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi081 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-14T02:25:58.472 INFO:teuthology.nuke:Checking targets against current locks 2023-10-14T02:25:58.506 DEBUG:teuthology.nuke:shortname: smithi081 2023-10-14T02:25:58.507 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T02:25:58.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-13 23:46:43.236397', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGpmrSRWtaEDZQ9e8lI3BFxZn+4PtOyoHSrzMszzz2knv/8eHKNBninwN6j44sm9RCFUamsrbYa94st1WdqRN4='} 2023-10-14T02:25:58.586 INFO:teuthology.orchestra.console.smithi081:Power off 2023-10-14T02:25:58.587 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T02:25:58.612 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2023-10-14T02:25:58.613 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:25:58.643 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-10-14T02:26:02.645 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:26:02.756 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-10-14T02:26:06.760 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:26:06.869 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2023-10-14T02:26:06.971 INFO:teuthology.orchestra.console.smithi081:Power off completed 2023-10-14T02:26:07.077 INFO:teuthology.run:Summary data: description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 9051.54066324234 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi081 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=51a70f87f6174c1ba21df61da8d90058 status: fail success: false 2023-10-14T02:26:07.078 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T02:26:07.148 INFO:teuthology.run:FAIL